System and method for 3-D projection and enhancements for interactivity

ABSTRACT

A system projects a user-viewable, computer-generated or -fed image, wherein a head-mounted projector is used to project an image onto a retro-reflective surface, so only the viewer can see the image. The projector is connected to a computer that contains software to create virtual 2-D and or 3-D images for viewing by the user. Further, one projector each is mounted on either side of the user&#39;s head, and, by choosing for example a retro angle of less than about 10 degrees, each eye can only see the image of one of the projectors at a give distance up to 3 meters, in this example, from the retro-reflective screen. The retro angle used may be reduced with larger viewing distance desired. These projectors use lasers to avoid the need for focusing, and in some cases there projectors use instead of lasers highly collimated LED light sources to avoid the need for focusing.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a U.S. National Phase of International PatentApplication No. PCT/US2011/054751 filed on Oct. 4, 2011, which is acontinuation of U.S. patent application Ser. No. 13/252,126 filed onOct. 3, 2011, which claims the benefit of U.S. Provisional PatentApplication No. 61/404,538 filed on Oct. 4, 2010, and also claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/516,242 filedon Mar. 30, 2011, the benefit of each of the foregoing is herebyclaimed. The subject matter of all of the foregoing is incorporatedherein by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to personal visualization devices, insome cases with 3-D and/or color.

2. Description of the Related Art

Currently, projection of 3-D video images requires bulky and expensiveprojection equipment, plus special glasses for the viewer and specialtheater-quality screens. Thus viewing 3-D videos is an expensive andspecialized experience.

As the world becomes more and more mobile, users want the ability totake all kinds of interaction with them. One phenomenon is the emergingcraze for tablets. The problem is defining the “right’ size, and howmuch connectivity to add. Also, there is the increasing cost with thesize of the screen as well as the weight, and the monetary loss shouldan unfortunate incident occur to such a device, like losing it, droppingit, spilling on it, etc.

Also, as more and more 3-D is available in places such as movietheaters, PCs, TVs, and home entertainment systems, users want thatexperience “to go,” but so far only a few devices are available,typically with very small screens using lenticular lenses. This approachrequires a very precise location, and it can cause all kinds ofundesired effects, to a degree that some manufacturers have substantialwarning labels, or outright do not recommend that young children usethem at all.

Also, currently even the most advanced premium venue-based stereo 3-Dprojection systems, such as Imax 3D™, are not capable of faithfully andexactly re-creating all the 3-D spatial clues required for eachindividual viewer's vantage point. Thus, such systems are essentiallyreducing the experience to the lowest common denominator. Furthermore,the one-size-fits-all “3-D” view does not allow for realistic motionparallax and other such strong spatial-awareness clues. More advancedexperimental systems that do try to accommodate such natural spatialclues require special additional eyewear optics, which tend to imposesevere restrictions on eye motion and field of view and suffer fromunacceptable image ego motion correction latency—causing visualdiscomfort, disorientation and nausea, and impeding natural interaction.

SUMMARY OF THE INVENTION

The present invention overcomes the limitations of the prior art byproviding enhanced systems and methods for 2-D and 3-D image projection.

One aspect of the invention is that it is lightweight, verypower-efficient, portable, relatively low-cost, and private, whichsystem may be used by one user only for viewing any type of audio/videopresentation, or by two or multiple users for conferencing,game-playing, teaching, and other, similar multi-person interactions.

Other aspects of the invention include a system and method offering veryinexpensive and even spill-proof or spill-resistant screens that arelight-weight and low-cost and can offer high-quality images on the go.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention has other advantages and features which will be morereadily apparent from the following detailed description of theinvention and the appended claims, when taken in conjunction with theaccompanying drawings, in which:

FIG. 1 shows an exemplary stereoscopic projection using aretro-reflective screen.

FIG. 2 shows a set of individual 3-D views rendered by stereoscopicprojection on a retro-reflective screen.

FIG. 3 shows examples of retro reflective surfaces formed by a cornercube embossed pattern.

FIG. 4 shows an example of retro-reflective microspheres embeddeddisplay surface.

FIG. 5 shows examples using stereoscopic projection to implement atelepresence multiparty video interface with good 3-D eye to eyealignment.

FIG. 6 shows an example of the power savings achievable with a fullyretro reflective surface.

FIG. 7 shows examples of viewing virtual images for a restricted numberof viewers.

FIG. 8 gives examples of the use of selective absorption filters orselective reflectors to suppress ambient light and increase imagecontrast in the field of view.

FIG. 9 shows examples of the use of fiducial markers in the screen todetermine the projector and the observer's viewpoint with respect to thescreen.

FIG. 10 shows retro reflective diffusion cone angle range requirements.

FIG. 11 shows examples of multiplayer games and 3-D interactionsurfaces.

FIG. 12 show examples of supporting laser pointing devices.

FIG. 13 shows an exemplary collaboration session.

FIG. 14 shows three screens, each with a different pattern.

FIG. 15 shows examples of a stereoscopic micro projector for in situ 3-Dimaging.

FIG. 16 shows two examples of structures that can be deployed to createretro-reflective surfaces.

FIG. 17 does not exist.

FIG. 18 shows two aspects of a personal viewer.

FIG. 19 shows aspects of projecting distinct pixel patterns duringinterleaving duty cycles, enabling capture and or creation of differentimages.

FIG. 20 shows two light sources and illuminating imaging array pixelapertures.

FIG. 21 shows how three primaries of each set may be combined, resultingin two complementary full color pixel patterns.

FIG. 22 shows each of the six primary sources projecting its own pixelpattern, which may partially or completely overlap on the screen.

FIG. 23 shows an exemplary retro-reflective surface.

FIG. 24 shows examples of placement and viewing of real and unreal 3-Dobjects correctly in the field of view.

FIG. 25 shows examples of various spherical retro-reflectors with“tunable” retro reflecting properties.

FIG. 26 shows examples of embedded fiducial screen patterns that enableinstantaneous determination of the scanning beam's position on thescreen.

FIG. 27 shows examples of embedded “cross hairs” fiducial screenpatterns that enable an efficient and instantaneous determination of thescreen position.

FIG. 28 shows various aspects of a conference in a telepresence system.

FIG. 29 shows an example of a dual-radius spherical retro-reflector.

FIG. 30 shows examples of detecting and adjusting for intraoculardistance and establishing consistent 3-D perspectives for multipleviewers.

FIG. 31 shows how displays on retro-reflective surfaces can be shown toa presenter in a manner invisible to the audience.

FIG. 32 shows “invisible” embedded retro-reflective fiducials.

FIG. 33 shows the optical divergence in the Z-axis of an object observedoutside the central view of a human and the effect of head rotation.

FIG. 34 shows detection of and compensation for head rotation to avoiddistortions occurring as a result of head movements in a stereoscopic3-D projection.

FIG. 35 is reproduced from Sony U.S. Pat. No. 6,956,322 B2, with somemodification to show a light-emitting device.

FIG. 36 shows a multi-primary engine with output.

FIG. 37 shows an exemplary Whisper Wall system.

FIG. 38 shows a refractive collimator and beam combiner for amulti-emitter diode stack.

FIG. 39 shows a wave guiding beam combiner system.

FIG. 40 shows the broad gamut of a five-primary system plotted in a CIE1931 2° standard observer chromaticity diagram.

FIG. 41 shows maximized efficiency using more than three visibleprimaries.

FIG. 42 shows another aspect of a Zoom-Macro augmented mobile visionfunction, enabling a viewer to use the system as a virtual microscope,enhancing or annotating real objects viewed through a transflectivescreen.

FIG. 43 shows examples of a reflective movable visor with projectors.

FIG. 44 shows examples of dual-projection systems embedded in thin eyewear.

FIG. 45 shows projection of “virtual hands” in stereo 3-D.

FIG. 46 shows examples of cubic retro reflectors with deliberate slightirregularities that improve the perceived image quality.

FIG. 47 shows a tiled configuration of altered type 1 and type 2retro-reflecting facets arranged into an array introducing deliberatedegree phase diversity.

The figures depict embodiments of the present invention for purposes ofillustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Stereoscopic 3-D Micro Projector

Two micro projectors each project one image from two separate positionsso that the image of each projector can only be seen by one eye.Stereoscopic images are projected with the left image and right imageprojected separately by each of the projectors. The images are projectedonto a retro-reflective screen surface, in such a way that the lightfrom each projector is primarily reflected back to the position of thatprojector. The left image projector is mounted close to the left eye;its light can only be seen by the left eye. Similarly a right imageprojector is mounted on the opposite side of the head and its lighttherefore can only be seen by the right eye.

A good example of the use of such a system would be for a mobile deviceuser, preferably a smart phone user, by mounting these two projectors one.g., a headset or glasses (or glasses frame) and placing or unfolding aretro-reflective surface (or “placemat”) on any work surface, such as acoffee table or desk or tray table. Now each 3-D image can be projectedin the visual space defined by that work surface.

The main advantages of such a system would be the following:

1. A display system of ultimate power efficiency. Since most of theprojected power is reflected back within a very narrow view cone, thesystem would therefore be easily over 10× more efficient than alreadyefficient pico projection. The result would be a luminous image within abroad field of view.

2. An unobstructed 3-D view, not requiring active or passive specialized3-D glasses, and hence compatible with the user's corrective eyewear.Also, no light losses introduced through active shutters, reduced dutycycles, etc.

3. A highly resolved 3-D image capability with a comfortable naturalfocal point. The images are projected at a comfortable distance, so theycan be easily resolved through natural accommodation without requiringmagnifying or reading glasses. This would make them much easier to useas compared to mobile displays that incorporate a lenticular surface(e.g., Nintendo DS 3-D). While the latter type displays also do notrequire glasses, they can be hard to use for older users withoutrequiring reading glasses, and they also typically severely limit theuseable range of head to device position.

4. A high degree of viewer privacy is ensured because the view isrestricted to a very narrow area of view (cone) of the user.

5. The ability to share a working surface, in this case theretro-reflective screen area, by multiple viewers with each having anindividual view in 3-D. Each viewer is provided with a differentperspective. For example, special finished conference room tables mayallow all users to have an area in front of them.

6. An improved social interaction during collaboration and playinggames. The participants can naturally be seated facing each other arounda coffee table or a conference table and maintain eye contact with eachother. Participants do not have to face screen a screen that blocks theview to others. They do not wear special eyewear obscuring their face.

7. The same screen can provide different images on the same screen (2-Dor 3-D) as required e.g., in a control center or operating room. Theseviews may or may not be shared by any or all participants depending onthe situation. (Viewers may “channel select” from multiple channels.)

8. An especially power efficient means of creating multiple 3-D views ona single screen. Since each view is provided in a small fraction of thetotal 3-D view space and each view only needs to be projected to thatspace, the total energy required to create the sum of all these views issmaller then the equivalent energy required for a shared standard 2-Dview. Typically, conventional 3-D methods require 3 to 6 times moreenergy than 2-D to achieve the same perceived luminosity.

9. Additional power efficiency can be realized by only illuminatingobjects to which the viewer directs his or her attention and suppressingor darkening the background.

10. When a large surface is used, the projection can be focused on wherethe viewer is actually directing his attention, thus a very largedocument can be read, or multiple documents can be opened and perusedsimultaneously by moving one's gaze toward various different positionsin the field of view. Furthermore, when attention is focused on acertain object it can be enlarged, or provided with additional contrastfor easier viewing.

11. People with vision problems could, for example, read a “newspaper”that is draped across the coffee table (actually, the special reflectivecloth is on the table), and the area they are reading would beautomatically enlarged. This would be independent of any 3-D features.Due to the retro-reflective aspects of the screen, a large view and avery bright image can be accomplished with a minimum of energy thusapproaching the efficiency of a retinal projection without therestriction of view or any of the other uncomfortable aspects of suchretinal projection. Also external light would not interfere. Due to thenarrow reflection angle only light sources directly behind the user'shead and pointing at the screen could interfere. A hat could provide inthose situations a “dark halo”, effectively blocking extraneousinterference.

12. Since typically only milliwatts of illumination are required, incase of a flying spot projector laser safety for the eye is more easilyaccomplished while still projecting exceptionally bright 3-D images.

13. Gaze tracking with unrestricted 3-D head movements enables a natural3-D motion parallax. The projectors' stereoscopic perspective can beused as a proxy for the direction of the gaze toward the screen and theviewed object on, in front of, behind, below or above it. By rapidlyadjusting the projected image in 3-D accordingly, the system canaccomplish a very natural motion-accurate 3-D parallax effect where, forexample, objects are realistically occluded when the head is moved whileviewing a 3-D scene at a relatively close distance.

14. The projector(s) can also contain 3-D accelerometers to track headmotion accurately. Additionally or alternatively, marks on thereflective material can be tracked by camera(s) mounted at theprojector, either or both in visible or invisible light (IR, UV).

15. Cameras or point sensor or line-scanning sensors can also be used tocorrect for brightness differences stemming from reflection differencesbased on angle of impact on screen surface. For simplicity, in thecontext of this document, referring to one shall include reference toany and all of them where applicable.

Screen Options

The application would require a retro-reflective projection surface.Such a surface is currently realizable using microscopic glass beads.The beads are very small: typically between 5 to 100μ, typically smallerthan an individual pixel. Each light beam hitting such a bead isprimarily reflected back toward the origin of the beam rather than in astandard reflection which would be specular and away from the origin ofthe beam at an angle equal to the incoming angle by the laws ofreflection.

Alternatively, a micro patterned structure can be embossed into a hardsurface—plastic or glass—creating a retro-reflective surface. Theso-called corner cube is a very efficient retro-reflector. A surfaceconsisting of millions of such tiny corner cubes would also act as aretro-reflecting surface. A corner cube pattern can be formed on the topsurface purely as a front mirror structure, or it can be embossed as arear mirror structure allowing for a more easily cleanable smooth topsurface which can contain additional filters or anti-reflective coatingsas desired.

FIG. 16 shows two examples of structures that can be deployed to createretro-reflective surfaces. They are typically partially embedded orcreated as part of a meta structure, which will be also discussedthroughout. For mobile applications it is conceivable to simply have a“placemat” arrangement where a cloth or scrollable material would berolled out or unfolded and draped on a hard surface such as a table. Theretro-reflective structures would be attached to, embedded into, orcreated as part of such a cloth or sheet material. Since a picoprojector or micro projector or nano projector based on scanning lasersor high efficiency collimated light emitting diodes (LEDs) would nothave a focal point, the angle and orientation and absolute flatness ofthe screen is not critical. In fact the projector would determine thelocation of the screen e.g., by edge fiducials, and the image-ifdesired-could be “attached” to the screen surface. That is, the imagecan be stabilized in a position referenced to the surface (moving ornot) independent of the relative motion between the viewer, theprojectors and the projection surface.

A good example would be that a newspaper could be projected onto aretro-reflective placemat in the exact position of the mat: either flaton the table, or, alternatively, the image of the paper could bepositioned (in 3-D if needed) at a more comfortable angle for theviewer, which would presumably be orthogonal to his or her view.

Alternatively, the retro-reflective surface itself might be angledorthogonal toward the viewer's gaze in the form of a foldable screen.Additionally, the retro-reflective screen might be shaped like anewspaper or magazine folded in two halves with optionally the foldingand unfolding action causing the pages to advance, as if reading aprinted newspaper.

Interactive User Interface (UI) Options

Interactive features are optionally enabled by feedback loops based onthe reflection of the scanning laser on the projection surface and basedon the disruption of that scan pattern, e.g., by tracking with earliermentioned cameras hands or fingers that would “turn the page of thenewspaper” as in the previous example.

An infrared or low-power strobe scan detects the screen surface. Theimage, for example, a newspaper—is “attached” to the screen. When thescreen is detected within the viewer's area of focus, the paper isprojected or the movies played. If the viewer is interrupted and looksaway from the screen, the projector detects the changed viewingcondition, projection ceases and the movie is paused. For “hands-free”operation based solely on gaze or the position or orientation of thehead, in situations such as, for example, for car navigation screens orin an operating room, this stop action could be important.

Projected 3-D objects can now be manipulated in 3-D. For example, whenthe light from the projection beams reflects off fingers, or by touchingthe object with a Magic Wand™, optionally with its own retro-reflectivestructures embedded so it is more easily located. The 3-D location ofsuch a Magic Wand, fingers or Magic Chopsticks™ can be determined bystandard stereoscopic analysis of the field of view—namely by comparingthe left and right image and extracting the 3-D field of view spatialcoordinates from the two views. (Magic Wand™ Magic Chopsticks™ and otherinput devices enabled by augmented reality capability of the proposedsystem are described in more detail elsewhere and in the drawings).Alternatively, pulsed infrared or visible signals can be used as ascanning laser or LIDAR. Accurate 3-D depth of field measurements can bedone using the phase or timing information derived from the reflectedlight.

In yet other cases, around a fiduciary marking, a small set of commandbuttons could be printed on the surface, or in yet other cases, buttonscould be projected at the “near edge” of the viewing field, allowing oneto “tap” on them, and the cameras (visible or invisible light) or inbeam reflection could pick up on that motion.

Additionally, structured light patterns can be projected by theprojectors and detected by a standard CMOS camera or by sensors (e.g., afast photo cell detector) mounted in line with or closely to theprojector mirror or lens detecting the scanning beam as it retroreflected back towards its point of departure (e.g., the scanning mirroror projection lens). Such patterns can also be used accurately todetermine the precise location of fingers or any other object reflectingthe projected pattern.

Additionally, the smart phone (or other, suitable electronic device)itself, typically equipped with accelerometers today, could be used as aphysical “magic wand” and waved in the air, flipped, turned etc asneeded to interact with the content and/or the application running togenerate the 3-D view.

Support for Laser Pointing Devices

In addition to the above means of interacting with the projected images,the screen characteristics can be modified to allow for the use of laserpointing devices with their output being detected by the cameras. Asdescribed elsewhere, the screen might be designed to be retro-reflectiveto the precise and narrow wavelengths of the primaries rendering thevisible image, and the screen might also (at the same time) be designedto reject all other stray visible light, so as to maintain good contrastof the projected images in an environment with ambient light. That samescreen can also (at the same time as providing the other two criticalscreen functions) be designed to diffusively scatter (& not absorb) aspecific wavelength (visible or invisible such as IR) used by thepointing device(s), so that the sensors or cameras that observe thescreen can register the position and orientation of the pointer'sreflections on the screen. For example, a small cross, projected, forexample, by adding a diffractive pattern generating filter to thepointer, would allow the camera to determine the exact origin of thepointing beam. An example is a small IR VCSEL output (850 nm). Such alaser could be a stand-alone pointing device or a tiny add-on insertedinto a mobile device, turning it into a pointer. The IR wavelengthoutput of such a pointing device, being neither a primary nor in theremainder of the visible range, would neither be retro-reflected back tothe pointer, nor absorbed (filtered out), but diffusively reflectede.g., by a separate layer in the screen.

When using a large screen, fiducial markings can be embedded in thescreen—such as crossed lines or crosshatch patterns. By using thesefiducials, the head position and vantage point (gaze) toward the screencan be determined. Therefore, a full 6 degrees of freedom can be used asthe basis for interaction with the view object. E.g., if an object isapproached, it can be scaled up or down to accurately reflect its 3-Dposition. When the viewer approaches the screen, a far-away object wouldnot change significantly in scale, whereas a close object becomessignificantly larger, resulting in a compelling 3-D perspective motionexperience.

If an object is fixed relative to the display surface then it can beviewed from all different angles and perspectives simply by walkingaround. This particular feature alone would make this 3-Dmicro-projection system the ideal collaboration tool or a fantastic,novel, virtual board game. The image would very much appear to be aholographic projection.

Telepresence

A relatively small retro-reflective screen surface can be large enoughfor multiple people to interact with a group of people at the same time,using the same surface. Each of the viewers or participants would seeother parties, and approximately life-size, through the screen in 3-D.When they look at someone they would see that person's facerealistically in 3-D and when another viewer would look at the sameperson they would also see that person's face in 3-D but the imageswould be different in terms of perspective, corresponding realisticallyto their particular position and perspective as it would be in a realmeeting. For instance if person A (he or she is an image viewed on thescreen) is looking at person B (a viewer in front of the screen) thenperson C (another viewer also in front of the screen) would not have eyecontact with person A.

A light, foldable screen, combined with miniature and lightweightprojector headsets, would enable a conveniently mobile multipartytelepresence system that a can be used anywhere, even in your localcorner coffee shop!

Virtual Camera Panning Enabled by a Reflecting Screen for TelepresenceApplications

Having realistic eye-to-eye contact is a key feature in providing a highquality conferencing experience, in fostering natural human interactionand in establishing trust.

It is highly desirable to have stereo cameras that follow and record thegaze of the speaker regardless of his or her head movements. This can berealized by shaping the screen into a spherical (or cylindrical) surfaceand designing the surface to be specularly reflective so that thecameras that are integrated in the head gear can record the reflectionof the user's face on the screen. This can be realized bystrobe-illuminating the screen with a light source that is sufficientlydifferent from the projection primaries so as NOT to be retro-reflected,but to be specularly reflected. E.g., if the RR beads selectivelyreflect the primary red—at 640 nm, e.g., by a narrow band Braggreflector coated on the beads' back surface—then an additional redillumination provided either by the same projector or by a separateillumination source (e.g., LED) at 660 nm would be specularly reflectedby a separate layer in the screen (e.g., in an otherwise transparent topcoating). By this method three separate illumination primaries could beadded, allowing for color facial video to be captured by the cameras.

Alternatively, a head-mounted white LED strobe would flash. The strobewould be positioned away from the eyes and the recording cameras. Oneportion of the light of the strobe would be retro-reflected entirelywithin a narrow cone back to the strobe, while another portion of thelight would be reflected by the screen, which then would illuminate theface, and after a second the reflection off the screen would be recordedas an image by each of the two cameras.

Furthermore, to enhance the acquisition of high quality images, theillumination could be strobed at short intervals and time-interlacedwith the projector output, so as to prevent interference. The camera isquite sensitive and might be shuttered at extremely short exposuressynchronized with the strobe. The additional illumination would notsignificantly reduce visibility of the screen.

The benefit of the above Virtual Panning Camera arrangement is that thescreen can be kept inexpensive, made from easy to recycle materials, andthat all electronics is kept confined to the headset, where it can bemultiplexed for other functions. The head position is not constrained tothe fixed camera angles, and there is no need to add a set of camerasfor every party online

A screen such as envisioned with multiple, wavelength specific,reflective properties could be printed with special inks on plain paperor some filmed, or pre-coated or optically laminated material. Glossy(specular), matte (diffusing), retro-reflective inks, and various colorfiltering inks are commercially available.

Consequently, this set-up would be very effective for a portableone-to-many telepresence system. Instead of a circular screen, atriptych screen with three separate views (three parties online) wouldbe realizable with only two cameras.

Additional cameras could be added to the head gear to provide views ofthe right and left side of the head, or this function might be realizedby using wide angle cameras.

Since CMOS cameras are tiny and inexpensive, adding cameras does notprohibitively increase the overall system weight, complexity or cost.

Ambient Light Suppression

This section describes various methods to suppress ambient light. Whenworking in ambient light the working surface/projection surface tends toreflect that light, thereby reducing the contrast of the image. Tocompensate, more light needs to be projected by the projector, requiringtherefore more power. This would tend to reduce the battery life of themobile device as well as increase the cost of the projector. Therefore,a high degree of ambient light suppression is desirable. One method isto add selective absorbers to the screen that would absorb most of thedaylight but not the specific wavelengths selected for the projector'sprimaries. Since typically the laser primaries have extremely narrowbandwidth (LED ones are slightly less narrow), only a very smallfraction of the visible spectrum needs to be reflected by the screen.One method for creating such a narrow bandwidth reflector would be tocoat the retro-reflective beads with selective absorbing dyes such thatonly the primary wavelengths are reflected. In some cases, for example,3 types of coatings are applied to beads, which then are mixed andapplied to the retro reflective surface, reflecting only specificwavelengths. Alternatively, the wavelength specific filters andreflectors can be added to the material in which the retro-reflectivesphere structures are made.

Alternatively, if a geometrically structured reflector surface is used,as described previously, the same absorbers can be added to thereflector material or separate filter layers can be added on top of it.The top surface of the screen would absorb all of the light except thespecific wavelengths of the primary which would be reflected by thestructure.

Alternatively, eyeglasses could be constructed to reject all of thelight that is not specifically of the wavelength of the primaries. Theseglasses would also function as regular sunglasses since most of thenormal full-spectrum daylight would be significantly attenuated. Notethat in such a case a high degree of privacy is ensured. The systemfavors a very narrow viewing angle shielding the view from others. Withsufficient ambient light the projected image would be invisible foranyone not using these glasses.

Estimation of Right and Left Separation

Assume that the projectors are mounted one inch away from the respectiveeyes and that the separation between right and left eye is approximately3 inches. With light reflecting from a distance of 30 inches, the angleθG between the “good ray” (which would be the one that goes into thecorrect eye) and the projector is approximately 2°. The angle θB towardthe other eye (“B” as in “bad” ray; the one that would cause cross talk)is approximately 8°. So the retro-reflective surface must be designed togive a slightly diffuse retro-reflection with a cone of 3 to 5°. Thisangle is specified as the observation angle, and for an entrance anglebetween 0° and 45°. The latter angle is the angle between the screensurface and the impinging rays. See the discussion of FIG. 10, below,for additional information, as well as throughout herein.

Note that for a homogeneous image an irregular response of an angledscreen (typical lower reflectivity at greater angles) can be compensatedfor by observing the intensity of the returning light and adjusting forit. (e.g., using a simple CMOS camera to monitor the returning image.)

Additional Cross-Talk Prevention and Compatibility with Existing 3-DMethods

In cases when the viewing distances are larger or when the retroreflective cone is not sufficiently narrow to prevent cross talk betweenright and left view, cross-talk can be prevented with existing meanssuch as active shutter eyewear, or by using opposite polarization in theright and left projector, or by using a dual set of primaries withslightly different wavelengths and passive left-right discriminatingfilters in the eyewear.

Further, in some cases, the software can be used to actively suppressparts of the image, creating a virtually increased nose acting as avision separator.

Switching Between 2-D and 3-D Views

The projector can also project a 2-D image. This image can be aligned tothe projection surface so that both the right and left projectorsproject an identical image or the 2-D image may be strictly coming fromone of the projectors.

E.g., when approaching a large projection surface, at some distance,objects are rendered 2-D, but as one approaches the objects are nowrendered 3-D. The system would be able to detect the viewing distanceand the screen distance and render the 2-D to 3-D transitions smoothly.

Field of View Specific 3-D/2-D View Adjustments

In some case where normally 3-D is not possible (e.g., in the peripheralvision far off center, there is no stereopsis) generating 3-D imageswould not be necessary or desirable. Therefore, with gaze tracking andtaking into account the relative spatial resolution limits, the renderedimages can be adjusted to and limited to 2-D in those areas.

This saves computer power; e.g., when one is not looking directly at oneof the parties in a multiparty telepresence session, that party does notneed to be rendered in 3-D. (See the description of FIG. 5b .)

An Ultra Light Version of the 3-D Stereoscopic Projector

The dual projectors can be attached to a headset, e.g., a stereophonicheadset with microphone, and have two small protruding projectors. Tofurther minimize the weight and size of the headset, the electronicspower supply, serial connection, and lasers or other light sources canbe packaged in a small clip-on device no bigger than a cell phone. Thisdevice would be connected to the headset via two optical fibers whichtransport the right and left modulated signals to the scanning mirrors.The projector on each side of the headset consists of a very smallscanning mirror, typically a MEMS device of less than 1 mm with a microlens (or micro lens array) which shines a collimated beam (or beams) onto the scanning mirror.

The Use of LEDs in a Flying Spot Projector

Instead of laser diodes primaries, colored LEDs can be used, because thehigh gain of the screen reduces the photon budget dramatically. WhileLEDs tend to be surface emitters with insufficient intensity (power persquare micron), at the low power level required, the LED dimension canbe kept small enough to allow their light to be collimated to asufficiently high degree for a flying spot projector of the typedescribed in this document. This approach can significantly reduce thecost of the system and allow the use of very small mirrors, as theenergy density is not as high. When low power LEDs are small enough theycan be directly coupled (“butt coupled”) without lenses to the core of afiber or a waveguide structure as part of the combiner optics guidingthe light to the scanning mirrors.

Clip-On Projectors (Added to Existing Eyewear)

Since a significant proportion of the population requires correctiveand/or protective eye wear, a “clip-on” option can further extend theusefulness of the proposed 3-D personal projection system. The addedadvantage is that this approach enables a smaller spatial offset betweenthe pupils and the projectors, so the eyes see reflected light closer tothe center of the cone of retro-reflection, where it tends to be ofgreater intensity. Narrowing the screen's retro-reflection cone allowsfor greater cross talk prevention and greater gain, lowering the powerrequirement to achieve the same luminosity. A dummy glasses frame can beused for people not wearing glasses.

Also, in some cases, ear buds may be part of these clip-on projectors,as well as microphones, providing 3-D sound for sending and receiving,if desired.

In some cases, a small box with a battery and a processor can be worn atthe belt, for example, with a cable going to the clip-on units. Thiscase could also contain video encoding and decoding, memory,non-volatile storage and Bluetooth or other suitable connections etc. soonly basic driving signals need to go out to the clip-ons or headset, tokeep the size and weight low. Alternatively to clip-ons, the projectorsand cameras can be attached to a headset-like device, worn over thehead, or over the neck like some headsets.

Two-Way Conferencing

By including more than just one camera, for example two cameras each onthe left and right side, an image of the user's head can be gleaned andstitched together, one camera for each eye and eye-area, and one camerafor each half of the mouth area, allowing to stitch together accuratelya “floating head” with correct movements and eye gaze. By using cameraswith uni-focus, for a desired range, no adjustment needs to be made forfocus, and algorithmic calculations can be made for the retro-reflectionshape and brightness distortions introduced.

Drawings

FIG. 1 shows an exemplary stereoscopic projection using aretro-reflective screen, according to the system and method disclosedherein.

In FIG. 1a , viewer 100 is looking at a retro reflective screen 107. Onthe left side of his head is a left view projector 101, which scans theentire screen with a multicolor laser beam 103. The retro reflectivescreen reflects the light from beam 103 and a narrow angular cone 105.Some of the reflected light reaches the left eye 106 but none of itreaches the right eye 108. The right eye 108 can only see imagesprojected by right view projector 102. The fully stereoscopic image isseen by the viewer.

In FIG. 1b the width of view is only bounded by the extent of thescreen. In every direction the reflected light from each of theprojectors is restricted to a very narrow viewable cone. Viewer 110 hasprojectors 111 and 112 mounted on the sides of her head. Projector 111scans a wide range on screen 117. For example, a collimated scan beam113 reaches on the left side of the retro reflecting screen. Some of thelight is reflected back (depicted as beam 119) exclusively to theviewer's left eye 116 as beam 121. Similarly, the same projector 111 canreach the right side of the screen with e.g., beam 120, reflected backto the left eye 116 as beam 121. So the left eye can see range 122, thefull sweep of the projector 111. However due to the narrowness of thediffusion cones 115 and 123 created by the retro reflective screen 117,the right eye 118 does not see any of the light created by the leftprojector 111.

FIG. 2 shows a set of individual 3-D views rendered by stereoscopicprojection on a retro-reflective screen.

Five viewers are in front of one screen 250.

Viewer 200 sees an object 205 projected on the surface of screen 250.The left eye 203 sees only the image created by the left projector 201and the right eye 204 sees only the image created by the right projector202. Because each eye sees the object 205 in exactly the same relativeposition, the depth perception in this instance is the same as anon-stereoscopic projection.

Viewer 210 sees an object 217 in 3-D at a distance z behind the screen250. The left projector 211 depicts a left view 215 of the object 217seen by the left eye 213. The right projector 212 projects a right view216 of object 217.

Viewer 220 sees a 3-D image of object 227 at a distance z′ in front ofscreen 250. Note that the image of the object on the right of the screen226 is rendered by projector 221 on the left and is seen by the left eye223, whereas the left image on screen 225 is created by the rightprojector 222 and seen by the right eye 224. Two viewers 230 and 240both see object 237 behind the screen, each seeing a full 3-D image ofobject 237 from their own individual perspective. Note that to createtwo stereoscopic images, four different images are projected on thescreen: 235, 236, 245 and 246.

FIG. 3 shows examples of retro reflective surfaces formed by a cornercube embossed pattern.

FIG. 3a contains an example of a corner cube 300 shown in two dimensionsonly. Rays 301, 303 and 305 are retro-reflected back in the samedirection they came from as rays 302, 304 and 306.

FIG. 3b contains an example of a corner cube structure 314 depicted in3-D. Ray 310 is reflected three times at points A 311, B 312 and C 313,returning in the same direction as ray 316.

FIG. 3c shows a side view of a retro reflective structure implemented asa corner cube shaped mirror surface on the back of a transparent screen.The acceptance angle of the screen is greater than 90° because prior toretro-reflecting on the mirror structure, impinging on the top surface321 of the structure, the rays are refracted toward the mirror at thesmaller angle than the angle of incidence due to the higher index ofrefraction of the screen material and Snell's law. Due to thisrefraction effect ray 324 is bents towards the reflector surface 320before being retro-reflected, and returns parallel to 324 as ray 325.

FIG. 4 shows an example of retro-reflective microspheres embeddeddisplay surface.

In FIG. 4a , microscopic beads 400 a-n are inserted in a surface 401.The material 401 is in itself reflective, such as aluminum, silver orany other metallic surface; or (as depicted) a separate reflective layer420 is added either onto the back of the beads or added to the surface.Ray 410 is first refracted into the glass bead toward the center in theback of the sphere, then reflected on the mirror surface 420 and exitsthe bead in the same direction that it came as ray 411. Similarly rays430 and 440 are retro reflected back as rays 431 and 441. It is shownthat the acceptance angle of such a display surface is approximately90°.

If the beads are not as closely spaced (as shown in FIG. 4b ) less ofthe light is reflected back, and more is absorbed by the surface, butthe acceptance angle 457 of the screen would be somewhat wider (up to120°, as shown). There is a trade-off between the width of theacceptance angle and the total reflectivity (the gain) of the screen. Anextra absorption coating 450 is added to the screen material 451, toabsorb any light 455 that does not get retro reflected by the beads.Optionally an additional reflective layer 460 undercoating the beads canbe added. This layer 460 might be selectively reflective only atspecific wavelengths so as only to reflect the projectors narrowbandprimary colors as described elsewhere.

FIG. 5 shows examples of using stereoscopic projection to implement atelepresence multiparty video interface with good 3D eye to eyealignment.

In FIG. 5a viewer 500 is looking at a retro-reflective screen 510. Onthe screen he sees the other party 501 in the form of a stereoscopicimage projected by projectors 514 and 515. Projector 514 generates ascanning beam 513. When the beam 513 arrives at position 508 it createsthe image of the eye 505 of party 501 as seen by the left eye 503 ofviewer 500. Viewer 500 thus has direct eye contact with party 501because the eyes 503 and 505 are precisely aligned. Simultaneouslyprojector 515 projects the right eye image of 505 at position 509,separate from position 508, so that the right eye 505 of party 501 is inexactly the correct 3-D perspective. Projector 515 also projects theleft eye 506 of party 501 at a position (not shown) so that the left eye506 of party 501 and the right eye 504 of party 500 are correctlyaligned with one another. Note that a second observer 502 does not haveeye contact with party 501, but sees his face in a realistic 3-Dperspective looking away from her. Observer 502 therefore knows thatparty 501 is addressing viewer 500, not her, as she would be able tonaturally observe in a real meeting. Four separate images are generatedand projected on the same screen at the same time and seen as twostereoscopic 3-D images without glasses. The total power required tocreate these four views with the four scanning laser projectors isminimized, because to render each view the laser projectors only have togenerate sufficient photons to image a single small object in anexceedingly narrow field of view (face to face, eye to eye). Optionally,the background can remain dark, and because it's the retro-reflectivenature, the entire screen reflection is directed at the narrow cone seenonly by each eye. Cameras embedded in the screen ensure eye contact.Cameras, either monocular or binocular, may be embedded in the screen atfixed positions, and images of parties in the conference can be lined upwith these cameras to ensure good eye contact. In the case of binocularview, the stereo camera positioned in the screen roughly aligns with theposition of each eye; in the monocular case the camera would bepositioned between the eyes.

FIG. 5b shows a person 520 having a telepresence session with a foldedscreen. The screen 524 is folded in a triptych manner with two folds 525and 526.

In the upper diagram, when person 520 looks straight ahead at person Cin the center panel, the projectors render images of persons B, C and Con the three panels. Image B is rendered only by his left projector 527.Image C is rendered by both projectors in 3-D, and image D is renderedonly by his right projector.

In the center diagram, when person A looks at the image of person B,person D is outside the field of view of person A and does not need tobe rendered.

In the lower diagram, similarly, when person A turns to person D on hisright, person B is not seen, and person C can only be seen in his leftfield of view in 2-D.

In FIG. 5c , stroboscopic projection and illumination multiplex thecameras and the display. In the top side view, projector 530 projects animage of party 538 on the screen 531. The viewer 538 looks directly atparty 539 and has eye contact.

In the center side view, a strobed illumination source 532 illuminatesthe screen area 533 faced by party 538. The light diffusely reflects offthe face of party 538.

In the lower side view, the camera 534 captures an image 538′ of theface of party 538, specularly reflected in the screen (the screen actingas a mirror). The strobe light illuminates the face only during darkintervals of the projector. The additional light source might be builtinto the screen, e.g., in the perimeter to stroboscopically illuminatethe face of the viewer, or alternatively the projector could generate avery short burst of white light (RGB) synchronously with the opening ofthe shutter in the camera.

The top view, at the bottom of FIG. 5c , shows an alternative approach,wherein the camera 533 (or dual cameras) can be placed just outside theretro-reflective cone 535 of projector 534, so the camera is not blindedby the projector's light.

FIG. 5d shows a virtual panning camera arrangement for a four-partytelepresence. A viewer 550 views three parties 551, 552, 553 on athree-segment screen (triptych screen) in a virtual panorama. Viewer 550is facing party 552 directly, and they have eye contact. Aforward-looking camera 556 (on the left side of the head of party 550)records facial view of party 550, as specularly reflected in the screen,with the screen acting as a mirror. This image is projected for party552, to be seen for his right eye (after reversing back the image so itis not a mirror view of party 550). The screen in this example might bea combination, such as a two-layer lamination, of two differentreflection surfaces. One layer would be a partial specular (mirror)surface for self recording and another would be a retro reflectingsurface for image projection. For example, the cameras capture theimages as specularly reflected in a partial mirror surface on a backsideof the screen, such as the normal-spectrum light from the viewer's face.The projector light, on the other hand, is retro reflected by structuresdescribed elsewhere, and here placed behind the partial mirror (which isnot or considerably less reflective to the projection lightswavelengths), in some cases, for example, by a retro reflecting surfacethat is specific to a narrow wavelength, and as such tuned to intercept(selectively reflected) and retro reflect only the narrow laser lightfrom projector's primaries. In some cases a reverse order might beemployed.

Another camera 558 mounted on the head of party 550 records thereflected image of the side of his head, to be seen by party 551.Similarly cameras 557 and 559 on the right side of the head of party 550record the left eye view of party 552 of party 550's view and party553's view of the right side of party 550's head. Each of the otherthree parties has the correct view of party 550. When party 550 turnshis head to another party, these views change correctly as if they weresitting in these actual positions facing each other.

FIG. 6 shows an example of the power savings achievable with a fullyretro reflective surface. FIGS. 6a and 6b show a comparison of a normalprojection screen with a retro-reflective screen.

FIG. 6a shows a normal screen 600. On such a screen, an incoming lightbeam 601, such as produced by a laser projector, reflects diffusely in ahemispherical shape 602. Such a diffuse reflection is called alambertian reflection. The wide cone, spread in 180 degrees, subtends asolid angle of 2π or 6.28 sr (steradian). Thus an observer 603 sees onlya very small fraction 604 of the overall reflected photons.

As shown in FIG. 6b , when a retro-reflective screen 610 is used, a beam611 is reflected in a very narrow cone 612. The cone 612 with a coneangle of α would subtend a solid angle of 2π (1−cos α). For example, forα=10° then the solid angle is 0.095 sr, which is only 1.5 percent of thesolid angle of a hemisphere.

In this case a significantly greater proportion 614 of the photons arereflected back directly at the eye 613 of the observer in close vicinityof projector 615. Therefore a retro reflective screen can be said tohave a very high gain, thus requiring less projection power to createthe equivalent luminous response as compared to a normal projectionscreen.

As an example: if the reflected light cone subtends an angle of 10° thengeometrically the image appears more than 50 times brighter than if itwas projected on a so-called standard lambertian surface: asurface-diffusing the light in a 180° hemisphere. It follows that ifsuch a narrow reflection angle can be achieved, a very considerableamount of power can be saved and consequently even 3-D projection isfeasible using a very small diode laser source, as it has sufficientillumination power. (Alternatively standard LEDs coupled into fiberswould provide sufficient power also, where greater coupling losses wouldbe offset by the greater efficiency of LEDs.)

FIG. 7 shows an exemplary situation wherein the system and methoddisclosed herein may be implemented for a restricted number of viewers.

In FIG. 7a , while playing a board game on a flat surface 702, player708 has a view 701 of the surface 702. Player 709 has a view 703 of thesame surface. 3-D images placed in view space 704 are exclusivelyvisible to player 708, similarly 3-D images in view space 705 areexclusively viewable to player 709. An example would be virtual objectsmanipulated by their hands in the spaces. Only objects placed in viewspace 706 can be shared, if so desired, but they also may be renderedexclusively viewable. In this drawing the projectors are not shown forobjects to be shown to both viewers. The image data have to be shared bysome means accessible to both projection systems, for instance anInternet accessible application.

In FIG. 7b , the screen 712 is mounted “planetarium style” on a surfaceabove both viewers. Viewer A sees images projected by his headset (notshown) in a cone shaped field of view 711. Viewer B sees a cone shapedfield of view 713. The two cones intersect, forming the boundaries ofshared field of view volume 716 in which A and B can share 3-D views.This shared view space expends to infinity behind the screen in volume717. The screen 712 need not be flat. Its retro-reflecting surface maybe of any shape, angular or smoothly curved, such as, for example, aconcave dome shape as in a planetarium.

FIG. 7c shows an airplane passenger viewing entertainment on a personalportable foldable screen or tray table 720. Passenger 721 is viewing a3-D movie on a triptych retro-reflective screen 722. Despites itscompact size, the screen's concave shape extends widely across his fieldof view and allows for ample head motion while observing in the 3-Dimages, such as the animated 3-D character 724. Due to the narrow viewangle of the returning light this movie and any images on the screen areexclusive to passenger 721 and cannot be seen by other passengers or aflight attendant coming by. If desired, a hat 723 provides additionalprivacy by creating a “dark halo” around his head.

In FIG. 7c the top view of the same arrangement shown in FIG. 7b clearlyshows that the triptych retro-reflective screen guarantees a “for youreyes only” view to passenger 731. Optionally, the tray table surface 736may also be part of the projection surface. Note again that any 3-Dimages can be rendered not only in front but also anywhere behind thesesurfaces, such as, for example, in the space 735. These 3-D view spacesvisually extend to infinity.

FIG. 8 gives examples of the use of selective absorption filters orselective reflectors to suppress ambient light and increase imagecontrast in the field of view.

FIG. 8a shows a sphere 800 embedded in a reflector material 805. Thesphere is covered with a coating 801 which slightly defuses the ray 806.This ray passes twice through this layer. Optionally the material 805 iscoated with an absorber material 804.

FIG. 8b shows a sphere 810 coated with a Bragg reflection coating 811.

FIG. 8c shows that after the sphere is embedded in the absorbingmaterial 815, the exposed surface covered with the Bragg reflectormaterial 812 is etched to remove that material, exposing the sphere,which is transparent. Ambient light such as ray 813, after entering thesphere, is not reflected and is absorbed in the absorption layer 815;whereas a primary ray such as 814 is reflected by the Bragg reflector,which is tuned to reflect this wavelength only.

FIG. 8d shows the effects when multiple color selective retro-reflectingbeads are mixed into one screen surface. Bundles of red (R and R′),green (G and G′), and blue (B and B′) rays impinge on three sphericalreflectors 820, 821, and 822, each of which have been coated toselectively reflect one of the primary wavelengths.

FIG. 8e shows that alternatively, spheres might be coated with Braggtype multiplayer reflectors that reflect a combination of the specificprimary wavelengths. In reality these multilayer periodic Braggreflector coatings are very thin, with a thickness 831 in the order ofmicrons, whereas the spheres' diameter 830 might be between 100 micronsto 800 microns.

In FIG. 8f , multiple retro-reflecting layers tuned to reflect each ofthe three or more primaries are coated on the spheres so that all of theprimaries would be reflected by each sphere. Three such coated spheres841, 842, and 843 are embedded in the surface 844, with the exposed topcoating surface removed by, for example, etching. These spheres reflectall three primary wavelengths (R, G and B). Note that the coating onthese spheres lets the other wavelengths pass through. That is, most ofthe ambient light spectrum is not reflected, but rather, it is absorbedin the layer below (absorption not depicted her, but shown previously inFIG. 8c ).

FIG. 9 shows examples of the use of fiducial markers in the screen.

In FIG. 9a , a rectangular screen 900 is depicted as viewed by aprojector camera 920 (shown in FIG. 9b ) positioned on an observer'shead. Corners 901, 902, 903 and 904 have fiducial marks or are simplylocated at the polygonal 4 corners of the retro-reflective structure.The center 905 of the screen 900 is defined by the intersection ofdiagonals 906 and 907. Since the dimensions of the screen are known inadvance, this is more than sufficient information to precisely determinein 3-D space the head position and orientation —in six degrees offreedom—vis-à-vis the surface of the screen and any location within thescreen, or relative to the screen in view.

FIG. 9b shows a side view in which observer 920 is looking at the screenwith corners 927 and 928. Projector 921 sends a beam 923 that reflectson corner fiducial 928, reflecting back beam 924 to cameras 922.

FIG. 9c shows a stereoscopic view of four fiducial markings 930, 931,932 and 933. The right camera 934 and left camera 935 each locate atleast three of the four fiducials. Each camera can determine its ownspatial position. Any objects scanned and detected by both cameras,whose positions can be estimated by means of 3-D stereoscopicperspective calculations, can now be located in a global coordinatesystem referenced in relation to the same fiducials.

FIG. 10 shows retro reflective diffusion cone angle range requirements.

To obtain an estimation of right and left angular separation of view,assume that projectors 1007 and 1008 are mounted one inch away from therespective left and right eyes 1001 and 1005, and that the separation dbetween the eyes is approximately 3 inches. The light from the leftprojector 1007 is reflecting on a retro-reflective surface 1000 at adistance D of 24 inches. The angle θG between the “good” ray 1003—whichwould be the one that goes into the correct eye 1000—and beam 1002 fromthe projector is approximately 2°. The angle θB toward the other eye (asin “bad” ray, the one that would cause cross talk) is approximately 7°.So the retro-reflective surface must be designed to give a slightlydiffuse but sufficient retro-reflection with a cone of 3° to 5°. Thisangle is specified as the observation angle. The screen should reflectthis narrow cone for any entrance angle (ideally) between 0° and 45°.The entrance angle is the angle between the screen surface and theimpinging rays.

FIG. 11 shows examples of multiplayer games and 3-D interactionsurfaces. Multiple participants each approach the retro-reflectivescreen surface. Their headsets determine their relative position withrespect to the surface and generate perspective-correct 3-D images ofthe scene to be viewed. Note that normally participants would becollaborating and they therefore would share the same view. However, itis possible for both parties to have a view that is partially shared andpartially separate, e.g., if a card game was played or a board game,some of the view would be jointly viewable by all, and some of the viewwould be private for each viewer to allow them to see cards or keeptrack of certain things in their field of view that would not be sharedwith other players, as also shown in FIG. 7 a.

FIG. 11a : Chess Action™. A 3-D chess game where player 1100 and 1101moves piece on a retro-reflective play surface 1102. Each player seesthe chess pieces 1104 a-n, of which pieces 1104 a and 1104 b are shownin this figure, in 3-D, from his or her side. The virtual chess piecesare moved by hand. Pieces may come to life when “touched” and completetheir own move. For instance, when touched, the knight 1104 may gallopto a new open position 1105 as intended. The projectors (not shown) ofeach player can render a full 3-D view of everything on, below or abovethe surface. Normally the 2-D chess board would be positioned on thesurface. However it is possible to envision multiple boards and verticalchess moves as imagined a long time ago in Star Trek as“tri-dimensional” chess.

FIG. 11b : Gulliver's Soccer™ (or baseball, etc). A soccer field 1116 isprojected onto a retro-reflective surface 1113 The players (such as1114) move realistically in full 3-D. All 3-D imaging is renderedsimultaneously and in real time by the viewer's dual headset projectors1110 and 1112. The viewers can choose to watch from different sides ofthe game as if in the stadium around the field. Each viewer has a full3-D view of the action on the turf. The projection surface 1113 needs tobe slightly larger than the field. The retro-reflective mat laid down ona coffee table would do the job. With fiducials embedded in that screenviewers can walk around the field. If they come closer they get aclose-up, and as they walk around the field they get different cameraviews in 3-D. This set-up can be applied both to real-life action gamesas well as simulated games rendered by 3-D graphics engines or acombination of the two. Optionally the observer may interact with thegame in progress by adding players from the sidelines (such depicted as1115) or by manipulating the ball 1117 with hands or special userinterface devices described elsewhere in the application.

FIG. 11c : Crystal Ball™. A crystal ball 1120 appears above a flattable-like surface consisting of a retro-reflective screen 1128. People1122 a-n, of whom 1122 a and 1122 b are shown here, arranged around thesurface see 3-D images appear in the crystal ball, as in a séance with asoothsayer. Note that four images 1123 a-d of the crystal ball arerendered on the projection surface. (In this drawing only threeprojectors 1129, 1130, and 1131 are shown.)

FIG. 11d : Magic Chopsticks™. In the black-and-white film classic“Musashi” the hero, played by the famous actor Toshiro Mifune, catches afly 1140 in mid-air with his chopsticks, thereby demonstrating hissupremacy as a swordsman and avoiding the unnecessary slaughter of adrunken trouble maker at the inn where he is staying. This is a seminalscene in this very famous movie. Magic Chopsticks™ are embedded withretro-reflectors and special fiducial markers so they can be projectedupon (become part of the screen) and also easily tracked in 3-D (furtherdescribed in the discussion of FIG. 11e ). These embeddedretro-reflectors may optionally be made not retro-reflective but opaquefor infrared, allowing them to be tracked invisibly. The chopsticks andother objects may also be located and tracked in 3-D simply by the dualshadows they leave in the retro reflective image. For example, a userworking above retroreflective surface 1146, as depicted, a chopstick1141 is simultaneously scanned by a left projection beam 1145 and aright projection beam 1144, leaving shadows 1142 and 1143, respectively,on the retro reflective surface. Here it is assumed that the location ofthe screen in the field of view has been previously accuratelydetermined, for instance by fiducial markers embedded in the screen orsimply by noting the corners and the rectangle's geometric distortion.

FIG. 11e shows how the Magic Chopsticks™ can be imaged holding a virtualobject, such as, for example, a rice ball 1152 with a natural lookingpartial occlusion of the real object behind it, in this case chopstick1151.

FIG. 11f shows the rice ball (and any other images) 1163 imagedpartially on the table surface that serves as a retro-reflectivebackground and in part directly on the retro-reflective surfaces of thechopsticks 1161 and 1162. Both chopsticks are located by fiducials (suchas those shown in FIG. 11g ) and “painted over” by the graphics softwareand projector illumination to make them (again) visible. Without thispainting over they would become substantially invisible. For chopstick1161 the surface area 1165 is now occluded by the part of the rice ballimage being projected directly onto its retro-reflective surface,whereas in the case of chopstick 1162 in the foreground area, 1164 ispart of the chopstick image (virtual object painted on top of realobject) occluding the rice ball image 1163.

FIG. 11g (inset) show a detail of chopstick 1162. The chopstick'ssurface 1166 is made retro-reflecting to visible light—the wavelengthsof primaries of the imaging system—but an IR black (IR light-absorbing)die stripe fiducial marking 1167 allows the scanner-detector system inthe headset to locate the chopstick 1162 precisely in its field of view.The MagicChopsticks™ game comes with sticks of exact known dimensions,facilitating localization.

FIG. 11 h: 3-D poker. Note the glasses 1170 shown in the drawing do notneed to have any optics. They are simply there to indicate the viewer'sperspective and the approximate position of his eyes and the cameras,projectors and any sensors in the headset. The virtual cards 1173 and1174 that the player is “holding” are shown to the player in his privateview space, as is the stack of his poker chips 1172. The cards 1173 and1174 are being played and are visible on the table 1175 to both players(in the shared view space). Real objects, such as a coffee mug 1176, canalso be on the table and do not interfere with the projection, as theydo not conflict in view space with imaged objects such as cards andmoney. The headset detects the position of mug 1176, for example, as astrong shadow obscuring the retro-reflective surface; and the game'ssoftware positions virtual objects in the remaining available space,avoiding a collision of real and virtual images. If required, suchobjects can be made part of the game.

Note that it is not necessary for the other players to be actuallyphysically present, and shown in the lower drawing. Each seat on thetable can be taken by a real person or remotely with the virtual imageof cards being dealt, just as if the person were present. The foldablescreen 1183 can be placed with its optionally in partially horizontaland partially vertical position so that virtual playerl 181 can be seenby real player 1182 in 3-D.

FIG. 12 show examples of supporting laser pointing devices. Four typesof screen light response are shown: FIG. 12a , diffused reflection; FIG.12b , absorption; FIG. 12c , specular reflection, and FIGS. 12d and 12e, retro-reflection.

In FIG. 12f , a diffusing layer 1240 consisting of microscopic particlesis combined with a retro-reflective back structure 1241. A beam of light1242 from a pointing device is diffusely reflected. Another beam (ofanother wavelength or polarization) of light 1243 passes through thediffusion layer 1240, retro-reflects on the retro-reflective backsurface 1241, and is reflected parallel the opposite direction as beam1244.

In FIG. 12g alternatively, the retro-reflective top surface consistingof partially embedded microspheres 1250 only retro-reflects rays 1251 ofthe narrowband wavelength of the primaries. Light of other wavelengths1253, such as from a pointing device, is diffusely reflected on a lowerback surface 1254.

FIG. 12h shows this phenomenon in some close-up detail. Note that whilethe microspheres' wavelength-selective, reflecting coating 1262 does notretro-reflect pointing device ray 1263, the spheres do help to furtherdiffuse the rays by optical refraction primarily on the exposedspherical air-surfaces

In FIG. 12i (side view), a camera detector 1279 on the viewer's headdetects a compass-like pointer image projected by a pointing device1270. In this side view only the points N (north), S (south), and C(center) along the vertical axis are depicted. As depicted in FIG. 12j ,below, the pointing device projects four cross points: N, S, E and W(north, south, east and west) at equal and known angular spread at angleα (alpha) from the center C of the cross. Mathematically the angle ofincidence of the center beam PC can be derived from the observed ratioNC/SC as observed in actual size on the screen. Because as previouslyshown the instantaneous position of projector/camera 1279 with respectto the screen is also known (assuming an a priori known screen geometryand screen fiducials or corner tracking) the observation of the crossNC/SC ratio and prior knowledge of the pointer spread angle α allows thesystem to accurately determine the inclination of the center of pointerbeams PC as well as the absolute position of pointer emitter P.

FIG. 12j shows a person 1280 manipulating a 3=D pointing device 1281 asdescribed above in the discussion of FIG. 12i . His headset detects (forexample, by a line scanning sensor or by camera 128) the positions of N,S, E and W on the screen 1282. As described in the previous section, thesoftware of his headset (or a connected game system) determines positionP of his pointer and the 3-D position of the center point axis PC withrespect to the screen. The dual projectors 1286 (left side) and 1287(right side) now can project a virtual pointer arrow 1288 in 3-D bypositioning a left image 1284 and a right image 1285 in the positionsand with the correct stereoscopic disparity for the viewer's perspectiveat that moment.

FIG. 12k shows that optionally, the position of the virtual (projected)arrow along axis PC can be manipulated by rotating the device or (asshown in FIG. 12l ) by a control such as a slider control 1292 on thepointer device 1293. This control may be connected to the headsetwirelessly by RF or by optical means, such as, for example, bymodulating the laser pointer output in such a way that it is easy todetect by the headset camera or optical sensor (photocell).

FIG. 13 shows an exemplary collaboration session. A participant 1300 ina collaboration session has a worktable with a retro-reflective surface1303.

FIG. 13a shows participant 1300 facing a second retro-reflectivevertical surface 1302. Participant 1300 sees an image of anotherparticipant 1311 at a remote location at position 1312 at the screen.This image is projected by his head set projector 1314 which also hasheadphones 1313 and microphone(s) 1315. When the participant looks downtoward the work surface 1303, in this position 1301 he sees the image ofan amphora 1305, as projected by his headgear in 3-D on both the worksurface 1303 and the screen 1302. Using a pointing device 1304,participant 1300 can modify the virtual amphora 1305 with a virtualbrush or carving tool 1316. Looking at the screen ahead, participant1300 sees the other participant 1311 and a virtual extension 1320 of thework surface 1303, so that the amphora 1305 appears positioned in themiddle of the joint virtual collaboration work space.

FIG. 13b shows this retro-reflective surface implemented as one foldablesurface system that is folded to a vertical position when a virtualcollaboration session is required with a participant at anotherlocation. The vertical screen section could be folded down like atabletop to modify the collaboration work area for a work session withthe local participant.

FIG. 14 shows three screens, each with a different pattern. The patternsare designed to allow a micro projector scanning sensor or camera todetect its relative position on the screen by detecting the fiducialpattern lines that cross its field of view.

In FIG. 14a the square grid pattern consists of horizontal lines 1401and vertical lines 1402 that are easily counted in a traditionalhorizontal and vertical scan pattern.

FIG. 14b alternatively shows a pattern of spaced dots 1411 in arectangular manner.

In FIG. 14c the diagonal pattern 1421 might be useful for a flying spotprojector. The projector's highest scanning speed is horizontal. Itwould always see the diagonal lines. By timing the detection of theselines, crossing the horizontal scan would allow the system to determinethe screen orientation (e.g., deviation from horizon) with respect tothe scan direction in the field of view.

It is possible to embed these patterns imperceptibly, e.g., by specificIR retro-reflector or absorber (shadow) patterns printed on the screensurface. Alternatively, the projector itself could create a structure oflight patterns that is “anchored” to the screen by fiducials or bydetecting the screen corners. For example, the left projector canproject such a pattern to be detected by the right camera or vice versa.

FIG. 15 shows examples of a stereoscopic micro projector for in situ 3-Dimaging.

In FIG. 15a an organ 1500 is a viewed by a surgeon wearing the dualprojectors 1501 and 1502. The projectors render two separate images 1503and 1504 on the surface of organ 1500. These are the left and right viewrespectively of a virtual object seen by the surgeon projected insidethe organ at position 1505.

In FIG. 15b a retro-reflective stain 1511 has been applied to thesurface of organ 1500.

FIG. 15c shows how this retro-reflective stain can be applied in twolayers on the organ 1523: 1) The top layer 1521 containsretro-reflecting structures such as a surface coating of exposedmicrospheres as described elsewhere, thus making the surface of theorgan into a retro-reflective projection screen (as noted elsewhere, thenon-flatness of the surface is of no import because the scanning laserprojection does not require a focal point), and 2) an undercoating layer1522 might consist of a non toxic spray-on reflective adhesive layer toadd reflectivity to the microstructure embedded in it.

In FIG. 15d , when the projector is turned off, the surgeon has a clearand unobstructed view of the organ.

Optionally, as shown in FIG. 15e , the 3-D image can be projected onto amist or spray 1546 containing small droplets that are retro-reflecting.Thus small intermittent puffs of such a mist from a nozzle 1545 createsa 3-D view inside the organ of the tumor 1540 by stereoscopic images1543 and 5044 retro-reflecting from the mist. The real view and thevirtual image follow each other intermittently and very quickly so thesurgeon can align his tools and, for instance, direct a biopsy needletoward a tumor inside the organ, which tumor has been previously mappedby scanning

Alternatively, as shown in FIG. 15f , a transparent retro-reflectivescreen 1560 can be positioned just above the organ 1562 in the surgeon'sline of sight, allowing images 1561 and 1563 to be superimposed in 3-Don the actual organ 1562 as viewed through the transparent screen.

In FIG. 15g , the surgeon 1570 aligns his biopsy needle 1571 toward apredetermined tumor location 1572 inside the organ 1573 provided in 3-Dimages 1574 projected on the screen 1575, rendering a realisticline-of-sight view 1576 inside the organ.

Eye Stalks

On one aspect, proposed is a novel approach for a personal viewer, whichcan deliver both simple and 3-D viewing, fulfilling all the marketrequirements at a very low cost and weight. FIG. 18 shows two exemplaryaspects of such a personal viewer, according to the system and methoddisclosed herein. First, it shows how each eye has its own image. Inparticular, with reference to the retro-reflected beams 1830 a (lefteye) and 1830 b (right eye), it is shown that the nose actually acts asa separator, so the left eye cannot see the retro-reflection for theright eye, and vice versa. Therefore, it is generally desirable keep theretro reflection cones from crossing over the nose. The four instancesof angle 1831 alpha shown in the drawing indicate the “opened up” (orspread of the) cone retro reflected by the retro reflective (RR)surface. Thus each eye can only see the image created by its respectiveprojector (located adjacent to that eye) due to a) the retro reflectionangle, which keeps the reflection shooting back toward the projector,adding in both directions the opening angle, which angle can becontrolled by tweaking the retro cube angles, or the sphere materialand/or buildup, as discussed later in the descriptions of FIGS. 25 and26 as well as other sections; and b) by the position and the lateraldistance of the viewer's eye from its projector on the one hand and thedistance from the screen on the other hand.

Eye Stalks are pods with an extra lateral offset of emitter-sensorsstrategically placed close to a user's eyes, which augment the user's UIwith the world. In some cases, they may be particularly useful when theview is a long distance from the screen. They include a set ofmicrophones, a set of cameras and/or light sensors, and a set ofprojectors. The Eye Stalks can be designed to wrap around the back ofthe head, over the ear (like some audio head sets), or over the headlike classical headphones (over the ear headphones), or yet in othercases they can be worn like eyeglasses. In some cases, they can alsocomprise two sets of clip-ons fastening to (or over) the ears orclipping to head gear (visor sweat band, cap, hat or other head wear) orany type of eyewear.

The scanning type projectors each find a common visual reference (usingcameras and or light return feedback from photocells) in the user'sfield of view, to align their projections and cross-reference theirimage.

Audio detection could be augmented by using microphone arrays that cansteer the audio foci, possibly aided by visual (3-D stereoscopic) inputsfrom the cameras or sensors (detecting mouth and lips of a person in thefield of view). Microphones could focus on the mouth (below) and asecond (or third) person's mouth (multiple foci are possible).

The Eye Stalks can be a flexible, goose neck style (as are little audiomike booms currently extended from headsets), forming a conduit forelectrical (high speed serial) and/or optical signaling (fiber) andpower for the devices in the tip of the stalk.

All the afore-mentioned sensors and the projector can be packed into atip having less than 3 mm cross section. Alignment and fit can be loose,because the feedback loops adjust the system's settings automatically.Head motion (relative to an object or surface in view) can be perfectlycompensated for—steadying the view in 3-D—and detected at the same time.In some cases, this ability to compensate for head motion might be usedas part of the UI system (head gestures, for example “yes” is indicatedby up and down motion and “no” is indicated by left to right to left,“next page” command is generated by right to left motion, etc)

Most of the electronics, combiner/mirror for optics, power, etc., can beelsewhere, for example, in a plug or clip on the viewer's body, in somecases using its own Internet connection, for example via WiFi, or inother cases, assuming that there is a wired or wireless connection (ofany suitable standard, including but not limited to, for example,Bluetooth, WiFi, ZigBee, serial port, proprietary port, USB, USB to go,etc.) to a smart phone that can offer local data and or Internetconnectivity.

Total electronic load (typically less than 250 mW) would be less thanthe requirements to back light screens in today's phones, so if the useof the device enables the backlight to be turned off (substituting forthe main visual interfaces) then the battery life of the smart phone isnot adversely impacted when the Eye Stalk device is plugged into thesmart phone host, and powered “parasitically.” If the Eye Stalks devicehas its own battery, the battery could be very small, light, andinexpensive, as it could be, for example, a standard phone-type lithiumbattery in a unit worn on the body and connected by cable, or in othercases more akin to those used in various Bluetooth headsets, etc. Evenwhen the projection, on a retro reflective (RR) surface presumably butnot necessarily, is not ON, the scanning beams, or at the very least theIR structural light projection can continue to project, in collaborationwith stereo cameras, and this continued operation can strongly augment anatural UI (for example, hand motion, gesture detection).

Simple things like scanning the environment for known references (walls,buildings, objects, and people) enable invoking context appropriately(also known as enhanced or augmented reality), then using the projectionto display that content on a suitable object, as it can be detected bythe cameras.

For example, the system could wake a user who has dozed off, in placessuch as the subway when the user arrives at his destination, or whenmeeting someone at an airport (face recognition). This system could behelpful for people with disabilities such as diminished memory oreyesight.

Mobile Augmented Vision

In another aspect of vision enhancement for the vision-impaired, ascreen—either RR or translucent RR visor—could be held, or flashedbriefly, in line with the natural field of view. The Eye Stalks couldthen project an image overlay with a view, such as an enlarged orotherwise visually augmented (image contrast enhanced) view of thenatural scene ahead. Such an augmented scene could be a combination ofthe high-resolution stereo 3-D view recorded by cameras in the EyeStalks, or elsewhere included in the headgear, combined with referencesfrom other sources. For example, maps or Google street view; real orsynthetic, may be super-imposed in a spatially correct manner on thenatural and video stereo 3-D images. This 3-D enhancement might alsoconsist of hyper-spectrally acquired imagery or LIDAR point cloud datamade visible by overlaid projection, either with a transflective RRscreen or by direct laser projection, for example, on close-rangeobjects.

Virtual Microscope

FIG. 42 shows in yet another aspect an exemplary Zoom-Macro function,enabling a viewer to use the system as a virtual microscope.

FIG. 42a shows long range vision system 4200. Viewer 4201 sees anotherperson 4202 approaching at some distance. His headset camera 4203 zoomsin, and then the magnified image 4205 is projected on the RR screen4206, whereupon he recognizes the person 4202.

FIG. 42b shows the microscope function of system 4200. An observer 4211sees a tiny insect on a retro-reflective examination surface 4216. Hisheadset cameras 4213R and 4213L zoom in on the insect, taking astereoscopic video that is projected by dual projectors 4214R and 4214Las an enlarged 3-D image 4215. Label arrow 4217 points at the locationof the actual insect 4212.

Camera Obscura Projection: Simple and Low-Cost

The enormous screen gain of a retro-reflective surface and concomitantincreased optical power efficiency enables the use of light sources ofminimal power and dimensions. A femto projection system of the systemand method disclosed herein requires less than 1 lumen in totalillumination power, as compared to 10-30 lumens required by picoprojectors such those as marketed by Microvision. For example, a 525 nmdirect green laser diode, such as recently announced by Sumitomo, whenused as the green primary in the system and method disclosed hereinwould need to produce less than 1 mW of energy (@500 lm/Watt. It wouldgenerate 0.5 green lumens, which would be more than enough to satisfythe green primary requirement of a projector nominally rated as capableof 1 RGB lumens. Such a 1 mW light source is very small. For example, alaser diode can emit such power from a facet with only 1 micron indiameter. The dimensional reduction enables a lensless projection, usinga pin hole instead of a lens, named “Camera Obscura” by the Germanastronomer Johannes Kepler in 1604. As long as enough light comesthrough the pin hole, a clear and sharp image is projected on a darksurface facing the pin hole. This approach implies a femto projectionsource such as a laser diode or a very small LED. In principle, such aprojector requires no other optics, just a pinhole source illuminatingan imaging device close by. The imaging device can be transmissive (suchas LCD or LCOS) or a mirror light valve (such as a DLP) or any otherimaging device.

Camera Obscura Projector with Spatially Multiplex Sources

In yet a further enhancement, one can illuminate the same imaging devicewith several pinhole light sources and project distinct pixel patternsduring interleaving duty cycles, enabling capture and or creation ofdifferent images. Each light source (s0, s1 and so forth) creates aseparate copy of the imager's pixel pattern (an image of the individuallight valves in the array) on the screen. For example, as shown in FIG.19a , using two sources (s0) 1901 and (s1) 1902 approximately one pixelwidth apart can effectively double the projected resolution on thescreen. The patterns 1904 (denoted as black pixels) and 1905 (denoted aswhite pixels) created by each source 1901 and 1902 and the same imagingdevice 1903 in alternating cycles can partially overlap or interlace.Two-point illumination sources 1901 and 1902 illuminate four pixelapertures in imaging array 1903 Time sequentially illumination sources1901 and 1902 are projected as eight separate pixels on the projectionsurface 1906. Furthermore, the sources can be of different colors anddifferent polarizations.

As shown in FIG. 19b , two sets of three primary R, G, B sources,comprise a total of six point illumination sources arranged in twocolumns 1911_0 and 1911_1, with each one red green and blue,illuminating a single pixel aperture 1912 of an imaging array 1914. Saidillumination sources arranged in columns 1911_0 and 1911_1 then projectsix distinct color pixels (1913R0, 1913G0, 1913B0, 1913R1, 1913G1 &1913B1) in two columns 1915_0 and 1915_1. The only requirement is thatthe imager must be fast enough to be able to temporally multiplexillumination by each light source sequentially, N illuminations perframe. Furthermore, the individual light sources can be modulated intime and intensity to conserve energy for darker frames or frames thatrequire less than full color. This spatial multiplexing results in thebest of both worlds: An inexpensive, relatively low-resolution imagingdevice can be used to render acceptable-resolution 3-D images, orhigh-resolution 2-D images.

Each of the multiple laser diodes sequentially turn on for very shortintervals, thus minimizing motion blur (low duty cycle and low holdtimes). FIG. 19c shows one full image frame timing diagram cycle 2101with six short sub frame pulses 2102R0, 2102G0, 2102B0, 2102R1, 2102G1and 2102B1. With a simple feedback mechanism from the screen, anadjusted pixel map is computed for each primary, resulting in maximumspatial and color image fidelity, preventing motion and color break-upand other artifacts. For example, a six-primary system can beconstructed with dual sets of RGB primaries with each set of oppositepolarization. The three primaries of each set may be combined, resultingin two complementary full color pixel patterns, as shown in FIG. 21.

FIG. 21a shows two interleaved pixel patterns 2110 aa-nn, consisting ofalternating columns, one set of columns resulting from a projection offirst source 0 and the other set of columns resulting from theprojection of a second source 1. FIG. 21b shows a similar patterninterleaving, but with sources s0 and s1 being polarized in oppositedimensions. FIG. 21c shows a checkerboard interleaving pattern,resulting, in this example, in offset odd and even pixel aperture rowsin the imager, so that the successive illumination patterns interleaveboth vertically and horizontally. FIG. 21d shows three illuminationsources offset in both x and y dimensions by one-half pixel distance,resulting in a partial overlap in both dimensions. The partial overlapmight be desirable to reduce spatial temporal aliasing artifacts. Insome cases, the described doubling of the resolution and/or themultiplexing of right and left images for stereoscopic 3-D imaging canalso be applied to a scanning spot projector, by adding a secondillumination position, such as, for example, offset vertically byone-half a pixel position, creating a line interleave pattern. FIG. 21eshows an example of a dual-axis scanning mirror 2151. On said mirror,two incident-collimated beams 2152 and 2153, emanating from two lightsources 2154 and 2155, reflect on said mirror 2151. The beams arecollimated by a pair of lenses 2156 and 2157. The resulting scan pattern2158 is formed by a series of interleaved lines 2159 and 2160.

FIG. 20 shows two light sources 2001 and 2002 illuminating imaging arraypixel apertures 2003, 2004 , and 2005. The projected pixels 2006, 2008,and 2010 are illuminated by source 2001, and the projected pixels 2007and 2009 are illuminated by source 2002. The later two pixels areinterleaved between the former three. By interleaving from numerouspixels a detailed image is rendered with effectively twice theresolution of the imaging device.

Combining each RBG set into one position would enable longer duty cyclewhite light projection (W=R+B+G). This arrangement would be, forexample, advantageous for reading documents in high resolution in brightdaylight. Alternatively, each of the six primary sources may project itsown pixel pattern, which may partially or completely overlap on thescreen, as shown in FIG. 22. Positions of the primaries can be organizedto overlap, to minimize color break-up or, alternatively, separate red,green, and blue positions can be used to support full color HD or abright black-and-white reader mode. In summary: A single low-resolutionimaging device, up to six low-power laser diodes, and a simpleretro-reflective screen enable a low-cost, efficient, high-brightness,versatile personal mobile projection system.

A Telepresence Communication Wall—“Whisper Wall”

A Whisper Wall is a means to connect work teams at separate,geographically remote locations, fostering spontaneous interactions andcollaboration. The whisper wall concept is the ultimate telepresencesolution, a continuous “teleportation wormhole” between two physicallyremote locations. This concept would be a great tool to bridge thecollaboration gap between two open office environments. Anyone, at anytime, could just walk up to the wall and look for a collaborator at theother side of the wall. They can dialog spontaneously and withoutdisturbing others on either side. There is no need to go to a separateroom (savings in real estate and office space are significant—all ittakes is dedicating a wall in the office). If desired, the “portal”collaboration interface can be extended to include tables, doors andcabinets. They just need to be covered by RR “wall paper” such as, forexample, the surfacing material made by Reflexite™.

FIG. 28a shows an exemplary four-party conference in a CollaborationCorner™ telepresence system 2800, according to one aspect of the systemand method disclosed herein. In this example three retro-reflectivesurfaces form a corner in which the viewer 2801 can see other remoteparties 2802, 2803, and 2804 sitting roughly at three opposite cornersof a conference table. RR surface 2801 is a table or desk-like worksurface in the horizontal plane. RR surfaces 2806 and 2807 are in thevertical plane roughly orthogonal to each other. RR surfaces 2805-7 thusform a cubic cone that can easily be accommodated anywhere in a modernoffice environment, for example, by adding RR surface treatment toexisting walls partitions and tables. Work materials and virtual objectscan be displayed anywhere in the view of viewer 2801, which is at least180 degrees wide in azimuth and 90 degrees in elevation.

FIG. 28b shows the top view of four parties in an exemplary four-partyconference, as denoted by the arrows. The parties 2813 and 2814 arelooking at party 2811, who is looking at party 2812. (For example, Jilland Jack watch John, who is presenting his work to his boss, Mary). Notethat each party may have the same unconstrained three-way view in ashared four-way collaboration space shared with three other sites. Eachlocation can have multiple participants, and each participant has his orher own individual, unique 3-D perspective of the shared workspace, witha choice of both shared or private view space. Sharing might also bewith any subset of the participants, regardless of location. Thus, theviewer can share with a remote participant but NOT with a localparticipant, so collaboration can go one-to-one ad hoc, as is naturalwhen taking someone aside briefly during a meeting, without undulydisrupting the workflow.

FIG. 28c shows a foldable, square retro-reflective projection surface2820 with a partial slit 2821. Surface 2820 has four sections and may befolded into an open cube (“corner cube”) by a three-step method. First,one half of section 2822 is folded up, and then second, all of section2822 is folded to the left. The third step is folding section 2823behind section 2822.

FIG. 28d shows how such a folded cubic corner screen may serve as alight-weight, portable conferencing tool for an ultra-connected mobile“road warrior” 2830 conferencing with three other parties 2831, 2832 and2833 projected on cubic corner RR surface 2834.

One other aspect of the invention could be a whisper wall system thatuses adjustable transflectivity retro cubes (electrochromatic or LCDtype) and adjustable transflectivity specular reflection, such asdescribed exemplarily in the discussion of FIG. 23 for conferencing,with time division multiplexing, in an arrangement of oriented RR cornercubes, in some cases with modified corners (90+/−alpha) for directionalspreading. RR surfaces can create a multi-user, multi 3-D view surroundaugmented reality (AR) environment used, for example, for entertainmentand training

FIG. 23 shows an exemplary retro-reflective surface according to oneaspect of the system and method disclosed herein, with adjustablereflectivity, using corner cubes. Cross-section 2300 shows corner cubesprotruding down as 2302. Partially reflective material 2300 ismetalized, with layer 2303 covering the corner cube material 2305.Electrochromatic material 2304, such as, for example, polymer dispersedliquid crystals (PDLC), is added as an immediate layer, enablingmaterial to have particular properties such that by applying voltage itcan be turned opaque, and by removing voltage it is then semi- or fullytransparent. Some materials have the opposite properties, whereby theybecome transparent when voltage is applied and opaque when voltage isremoved. Changing the opacity of the background of the mirror in thefully transparent mode gives the viewer a partial view through thesystem, indicated by entering light beam 2301 a, which is split at thesemi-mirror into beam 2301 b retro reflected and beam 2301 c goingthrough the system. The beam is split accordingly. Turning on theopacity of layer 2304 lets “almost no light” (none for all practicalmatters) exit as beam 2301 c, and the majority of the light is reflectedas beam 2301 b. Most of these opacity systems turn whitish, indicatinghigher reflectivity, which helps turn the semi-mirror into a fullmirror. A semi-mirror need not split the light exactly 50:50(transmissive:reflective); the split ratio may be as uneven as 5:95, orany usable range. The voltage could be applied between, for example, theoptical filling block 2306, which is used to counter-fill the retroreflective corner cavities and the semi-mirror by applying voltage 2307.Depending on the type of PDLC or other material, either ac or dc currentis required, and either low or, in some cases, very high voltage for theelectric field. Some materials allow a gradual control, enabling, forexample, a window in an office to become opaque in lieu of curtains, soit can be used as an RR screen. This transparency variability can beachieved by either time multiplexing or variable voltage. These layersmay be applied, for example, by vacuum deposition of the partiallyreflective material in the appropriate thickness, and then spraying orother methods for applying PDLC on top of material before filling inwith inert resin 2306 as a protection and to create a flat opticalsurface on the other side. Additional uses of this variable transparencyRR material could be, for example, including but not limited to cubiclewalls, car windows, etc.

Visual information can include a full eyeball-to-eyeball mode withcorrectly aligned cameras, accomplished by reserving certain locationsand just embedding extra cameras in the wall for this purpose. Moresophisticated methods of interaction might include moving cameras thatfollow the person approaching the walls or a semi-transparent mirroreddisplay (similar to a Teleprompter) that enables the cameras to behidden and fully aligned. A third method to achieve correct gazealignment between two parties is to use the field-sequential nature ofthe projection to record camera images through a semitransparent displayusing strobe lighting that lights the person in front of the camera onlywhen the display is blacked out. Assuming the camera only uses visiblelight, this technique may be used when IR data is being broadcast, forexample, for audio, and/or concurrent with other IR beaconing operationsbeing performed by the system. Exemplary purposes of such a techniquemay be to locate the viewer's instantaneous head and gaze positions.

FIG. 37 shows an exemplary Whisper Wall system 3700 according one aspectof the system and method disclosed here. Each site has a group workarea: cubes, bean bags, Fuss ball—a typical open office type environmentthat stimulates spontaneous brainstorms and ad-hoc collaboration. Onewall in each site is entirely a large continuous screen. It provides aview of the other site as if it extends from the video wall 3701 onward.Team members wear some kind of wireless projection headset 3702 that isalso equipped with the cameras previously described. When they want tocommunicate with colleagues at the other site all they have to do isapproach the wall 3703. When they do, their image 3704 (real time video)increases on the screen in the other site as they approach the wall. Thewall has built in cameras, for example, hidden peep whole cameras 3705and 3706, or the screen is semi transparent and some kind of opticalmultiplexing (such as the variable transparency discussed herein). Thecamera views are seamlessly stitched together in a large relativelyundistorted view of the whole area work area. Each headset has amicrophone 3708 and 3707, and as the cameras 3705 and 3706 pick up aperson approaching, that person is identified and the microphone isswitched on, the audio stream is captured and transmitted as part of theimage transmission to the other site. The image of the person may have atext balloon 3709 (for example, when muted) or other visual indicatorthat says audio is available. As a person in the other locationapproaches their end of the wall the same process happens on the otherside. As they approach each other, the audio data is picked up by eachof the headsets from the IR data pattern 3710 being broadcast, invisiblyinterleaved in the image field and detected and decoded by thecamera/projector headset 3712. The person's image (on the wall) would bethe preferred location for that data audio field. In this way each canhave an undisturbed impromptu one-on-one meeting. Others can join bysimply approaching. The audio is only played on the headsets so othersin the work area are not disturbed by speakers. Ambient sound is nevermixed in since all voices are recorded very close to the mouth. Naturalor even whisper level talk is sufficient. Communications are directedand clear without any ambiguity about who said what still prevalent inteleconferencing today. Conversations can merge and side conversationscan develop spontaneously. People can walk away, splitting off from themain conversation, and return right after by just moving a few feet.Direction of gaze 3711 and relative position determines who can hearwhom and how much. Some kind of enhanced privacy mode is possible ifrequired (one-on-one locked, muting others) or can be enforced in aprivate talk zone (for example, in the corner).

Reflective (Polarizing or Standard Half Mirror) Movable Visor withProjectors

A convenient alternative to “glasses,” exploiting a perfect (narrowcone) retro reflective screen and also a precisely aligned optical pathto both create and observe the view from the exact eye position. A visor(e.g. baseball cap) gives shade (when folded out of view) and looks“normal’ in that position.

FIG. 43a shows a baseball cap 4301 with a shade visor 4302 from which a(polarizing or half) mirror is lowered when a projection is needed.Micro projector 4304 is embedded in the visor upper half and projects ascanning beam downward. After reflecting on the mirror, the scanningbeam creates an image on the retro reflective screen 4305. Additionalparts of the visor system are earphone 4306 and microphone 4307.

FIG. 43b shows that when not in use a second visor 4313 may be foldedout of view under shade visor 4312.

As shown in FIG. 43c , mirror 4323, when in use, can be adjusted(optionally automatically) up or down to align the projection in themost comfortable viewing position. Note that this arrangement also maybe advantageously implemented for non-stereoscopic viewing (with asingle projection for both eyes), using a regular (non scanning)projector. Optionally, the whole mobile system, including smart phonefunctions, may be integrated into this type of headgear, which can alsofunction to keep the head cool and shaded. Such a fully integratedsystem might be styled substantially similar to existing headgear suchas a sports helmet or baseball cap, facilitating user comfort andenabling early adoption of the new technology.

Use of Auto Focus Mechanism for Converging Scanning Beam (AutomaticAdjustment During Sweep)

When using an LED it may be desirable to have a scanning beam that is“over collimated,” into a converging beam. This approach enables a largeamount of light from the LED to be used in forming the image. Increasedprojection efficiency can mitigate the effect of the etendue (as used inthe field of optics) limitation of LEDs. Thus there is a focal point orfocal radiance distance where minimum spot size=resolution maximum isachieved. In such a case, detecting the actual RR screen distance duringthe sweep (scan) and adjusting the focal point is potentially of value.The beam convergence point can be adjusted by moving the lens or mirrorsurface, by changing the lens or mirror focus, or by moving the lightsource by a means, for example, of a fast piezo), or by some combinationof all three methods. If multiple lenses or mirrors are used, such amethod might be part of an auto focus system. Note that when a flatsurface is scanned with a moving flat mirror, some part of the screen isgoing to be out of focus (assuming an LED pixel sequential scan).

Use of a Retina and/or Iris Scan to Secure Data

A (second) half mirror can be used to look back at the eye (or iris), inaddition to projecting forward, so the low-power scan mirror would scanacross the retina and the reflected light would be detected, forexample, by a camera or photocell.

3-D Overlay: Recognizing 3-D Structure in View (Real and/or Projected)

In 3-D augmented reality (AR) space there is a significant challenge inhow to place an advertisement, and how to mix, combine, and merge realand unreal 3-D objects correctly in the field of view. There is greatcommercial value, such as the value of Google advertising, inanticipating and solving the problems that may occur: For example, theadded image inserted should not obstruct the view of key objects in theview (this would be annoying and potentially dangerous). These keyobjects in the view can be both projected and REAL. The personal viewingsystem described in this disclosure “mines” currently available FREEPROJECTABLE space in the viewer's gaze. The system scans for screenspace and margin to permit some head motion AND not violate 3-D humanfactor rules (right disparity, avoid frame violations). In the case thatthe views are all projected (no AR, no cups or objects in front of thescreen and no hands) then the disparity can be adjusted (Z depthadjusted) and room can be created for a banner ad or a cute object toattract attention (hence a dwarf or Smurf hailing the viewer). In thecase that both real objects and projected virtual objects are in thecurrent view, the system detects the real objects, such as hands infront of the screen or a cup standing on the screen, and adjusts (movestheir position in available the 3-D view space) only those virtual itemsthat are not directly related to the real items. For example, in FIG.24, described below, the animated character 2403 must remain in same 3-Dposition standing on the rim of the real coffee cup 2402.)

FIG. 24 shows a system 2400 according to one aspect of the system andmethod disclosed herein. A user 2410 sits at a table with a retroreflective surface 2401, which enables him to read a virtual newspaper,browse the web, or engage in any other, similar activity. On surface2401 is a cup 2402, which the system senses because the cup blocks outretro reflectivity in this section of surface 2401. As the viewingaspect is known through the position on the screen and the part that theuser can view, a virtual image during a subsection of the scan 2407 iscreated to appear on top of his coffee cup 2302. This image 2403 couldbe, for example, a dancing mascot of a competing coffee company makingcompetitive offers, generated from data showing where the user islocated, determined by means such as GPS location technology from hissmart phone. Camera system 2405 is head-mounted; and full scan range2406 covers a larger area of the table than just the cup, enabling theuser to read his virtual newspaper 2411. The viewer would interact withthe newspaper in a very natural way, by pulling pages across. The camerain unit 2405 can see the interception of his hand in the beam, analyzethe motion, and react to it as a touch screen would. Because thestereoscopic location of his hand may be determined by combining the twoimages of the left and right projector and camera system, the systemalso knows whether his hand motion applies to the surface of virtualnewspaper 2411, or whether his is simply reaching for his coffee cup.

FIG. 25a shows a retro reflective system 2500 made of multi-densityspheres, according to one aspect of the system and method disclosedherein. Beam 2501 enters the sphere, is deflected through a differentsphere, and comes out with a deviation alpha as beam 2502. By changingthe sizes and ratio of the respective refractive indices of materials n12506 and n2 2507 (n refers to the refracting index of the opticalmaterial), as well as the size and extent of the reflective section2505, the sphere 2503 may be used to create different cones. It isembedded in an RR surface 2504.

FIG. 25b shows an RR surface 2504, an incident beam 2501, and resultinghollow cone 2502. Cone 2502 is a conical, hollow cone creating a“doughnut” of light at the retro reflective surface. By adjusting theangle of the RR opening, more light is reflected toward the diffusionangle matching the location of the nearest eye, but not so far as reachthe other, more distant, eye, so potential stereo vision cross-talk isminimized. For example a surface embedded with the followingconcentrically layered spheres creates the desired “hollow cone” or“doughnut” reflections as follows: The spheres are “concentric shell”retro-reflecting microspheres of typically less than 1 mm diameter (lessthan the pixel size). The inner radius is 6/10 of the total radius. Theouter material has a refraction index of 2.95, which is relatively highbut available (in, for example, specialty optical glasses) and the innersphere is of a lower index of refraction index of 2.2. The advantage ofthis arrangement is that rays impinging on the sphere at relativelyoblique angles are redirected toward the bull's eye (the direct centerof the sphere as seen from the direction of the incoming ray) and thusthe reflected beam is corrected in a direction more parallel than itwould otherwise be without the lower index core.

FIG. 25c shows a sphere entirely made from a transparent material, suchglass or plastic. Two rays are shown. The first ray 2510 impinges on thesphere's surface 2511 at ½R (the radius) the distance from its centerline (the center line is the line parallel to the direction of theincoming ray through the center C of the sphere). The incident ray 2510impinges on the surface 2511 at 30 degrees from the normal. If the indexof refraction of the material is 1.932, rays impinging at 30 degrees areprecisely refracted as ray 2512 toward point B (bull's eye) on thecenterline, and then reflected back by a reflective coating 2513 as ray2514 to the front. After another refraction, the retro-reflected ray2515 exits exactly in the opposite direction and parallel to theincoming ray 2510. A second ray 2516 is shown impinging at 53.1 degrees,which is the angle of a ray that impinges at a distance of 80 percent ofthe full radius. For a refraction index of 1.932, the reflected ray 2717is 8.4 degrees diffused away from the incoming direction, creating arelatively wide cone that, for greater viewing distances, can causesundesirable levels of cross talk, hindering stereopsis. This cross talkis caused by the outer edge of the reflected cone becoming visible tothe other “wrong” eye (for which the image was not intended). In anear-eye/no-glasses-required projector arrangement as envisioned, it isdesirable to both maximize the light toward the outer cone ofapproximately the angle and at the same time cut off or redirect anylight that would create such crosstalk.

FIG. 25d shows the same configuration with a higher index material(N−=2.95). Rays 2520, 2521 and 2522 impinge at 17.5, 30 and 53.1 degreesrespectively, hit at 5.8, 10.5 and 21.7 degrees below the centerline B(bull's eye), and are reflected back as rays 2530 2531 and 2532, withdiffusion angles (variance from what would be a perfectretro-reflection) of 11.5, 21 and 43.3 degrees, respectively. Clearlythe sphere in C is more acting like a diffuser and not much as a retroreflector.

In FIG. 25e a second inner sphere 2540 (dashed line) is added with alower index of refraction (N2=2.2). This inner sphere 2540, byredirecting the outer rays, limits the diffusion angle and thus preventsthem from causing crosstalk. In this example the ray 2543 impinging at80 percent radius (53.1 degrees) would only be diffused 1.6 percent,exiting the concentric spherical reflector as ray 2553. Thus such aconcentric-shell, RR-spheres arrangement (R1/R2=0.6, N1=2.95, N2=2.2)limits the diffusion angles to within 2.4 degrees and therefore can workwell for distances up to 72 inches, assuming an inter-pupilary distanceof at least 2.5 inches (typical for adults).

Note that microspheres with a suitably high N of around 1.80-1.85 canfunction as an acceptable element to a retro-reflective screen to beused at short distances (less than 1000 mm).

FIG. 25f shows a good “donut” shape cone diffusion for a simple spherereflector. The graph and calculations of FIG. 25f show that for adiffraction index of N=1.82, more than ⅔ of the reflected light fallswithin a 4-degree angle.

FIG. 26a shows an exemplary system 2600 according to one aspect of thesystem and method disclosed herein. System 2600 may be used for scanningan RR surface and using the reflection, for example with an additionalinfrared beam that is deflected, together with visible light. Bychecking for presence or absence of reflection, the extent of thereflective surface area may be scanned. Additionally, in some casesmarkings may be added to further augment the ability to locate the beamat any given time. Infrared light source 2602 is sending out a beam viamirror 2603 that scans a surface line 2607 b. The scanning trace 2608 aof the IR beam is shown here. Also shown are intercept points 2607 a1-an at the edge of the material, which are the points where theretro-reflection starts, and points 2607 b 1-bn, where it ends. Thus theextent of the surface may be determined. Even though the scanning inthis example is shown going from left to right, it is clear that thescanning may proceed in either direction and in any suitable pattern,such as, for example, a zigzag pattern or a Lissajous figure pattern. Itis also clear that the geometry of the beam may be corrected with helpof transformative calculations (2-D and or 3-D) to deduce the size ofthe available RR surface in the system. In some cases, more than onesurface may be available. As the retro-reflected beam travels back onitself, an infrared receiver may be posted either next to light source2604 or just adjacent to mirror 2605. An infrared narrow-band filterreduces the incidence of daylight. Additionally, the infrared source maybe pulse-modulated to indicate pulses and increase the energy withoutincreasing the overall consumption of energy. The markings 2610 a-n maybe printed, for example, in infrared-visible only ink, which is ink thatis almost completely transparent for normal light, but turns black ininfrared light. Those may be used to print additional markings orstripes. In this example, stripes are printed that contain two sections:ID and location information that can say, for example, “I'm panel numbern, and I'm positioned 1 inch from the left edge,” and then a secondsection that states the distance from the top edge of the RR surface.For example, in FIG. 26b , the beam 2608 crosses the section printedwith a 3 (it would be in bar code, not in numerals) indicating thatsection is 3 cm from the top. An ID section on the left, for example,would contain the ID, so each surface has a unique ID. This sectioncould also contain copyrights, etc., so generic surfaces could not beused by turning off the visible light if a marking is not according tothe code, etc. Certificates and other individual markings could also beembedded. By having the same beam travel across multiple stripes, evenwhen held crookedly the system can orient itself and can, on each singlebeam trace, calculate the full extent and position of the surface andadjust, for example, the image, so that when a user waves with a panel,it would act like a newspaper, rather than like a projector, where thescreen would move, but the projected image would remain attached to thescreen (as real ink is fixed to the surface of a real paper). In somecases, the tracking screen position, orientation, and objects can bedone with a single photo sensor and a low-cost projector, detecting RRedge contrast during a sweep. In some case, rather than just a singlesensor, a small black-and-white (IR) sensor array (2-D or 1-D array) canbe used as well for faster or more accurate detection.

FIG. 27a shows an exemplary system 2700 according to one aspect of thesystem and method disclosed herein. System 2700 has an RR surface 2701marked with an IR light-absorbing cross-hair, a horizontal band 2702 anda vertical band 2703. Initially system 2700 scans in wide, broadlyspaced scan lines, a pattern optimized to quickly and efficientlydiscover the rough position of a projection screen within theprojector's operating range. Rather than continuous lines, the discoverypattern 2705 might comprise evenly spaced “pin pricks” consisting ofultra short duration laser diode pulses of e.g. Near Infra Red (NIR)laser with a wavelength of 850 nm. Such pulses might be spatiallysparse, but of just sufficient intensity and frequency to guaranteedetectable RR return pulses in the presence of the high-gain RR surface2701 anywhere in the projectors' scan range. Once the screen surface2701 has been detected, its rough center position, dimensions, andorientation are estimated from markings such as, for example, the crossbands 2303-5 or other fiducial markings.

FIG. 27b shows a successive operational phase of system 2700, but nowthe scan pattern 2715 has been narrowed down to an area just greaterthan the contours of surface 2711. The vertical scan angle (rangeangular scan) of the deflection mirror (not shown) has been reduced sothat the number of lines scanning on the surface can be increased whilemaintaining or even increasing the frame rate. Similarly, the horizontalscan angle range is reduced to shorten the “overshoot” on either side ofscreen 2715, or, alternatively, the screen discovery beam is turned ononly during a subset of the full scan width, thus reducing the line onthe time duty cycle. The second option might be preferred when aresonance type beam deflector is used.

FIG. 27c shows how once the relative position orientation of the screenwith respect to the head-mounted projectors has been fine-tuned, in anoptional third phase, the IR scan pattern might only be on very brieflyin the center of the cross bands to re-verify their position. As shown,just a few short (truncated) scan pulses 2725 at the expected locationconfirm the presence of diamond shaped center fiducial 2724.

FIG. 27d shows an alternative approach, wherein, the screen positionrelative to the projectors has changed, as indicated by the change ofthe center fiducial from the expected (previous) position 2735 to a newposition 2736.

Thus by such an escalating multi-phase scan procedure, system 2700 candiscover and track changes, and carefully maintain a real-time trace onthe exact location of the screen, while expending minimal energy in theprocess.

In many cases, a full motion 3-D user interface providing a naturalinteraction with virtual objects may have many 3-D objects (flashing 3-Dicons, buttons, animated vectors, or action figures) projectedintermittently against a dark background. In such an environment, havingan accurate and up-to-date map of 3-D space as outlined above is animportant part of enabling a highly versatile, efficient, always-onmobile UI. Such a system also guarantees that all objects are alwayscorrectly projected in the available 3-D space and with correct relativepositions and occlusions for the viewer's particular instantaneous,unconstrained vantage point (see a discussion of adjusting forindividual human factors in: “One size does not fit all”)

Folded TIR and Polarizing Splitter for Projection Glasses with ThinnerGlasses

If eyewear is acceptable, the dual-projection system can be integratedand a high degree of alignment can be achieved between the projectionpath and the eye, allowing for a stronger screen gain delivered byreflecting on an RR screen with a retro-reflection cone diffusion angle,using, for example, retro-reflective materials, such as Reflexite withnarrower angles. This approach enables a greater screen distance withoutcross talk, lower power consumption, greater privacy, and a morecontrolled view angle, thus limiting unwanted shadows and occlusionswhen objects are in the path of the dual projectors. In addition, theprojectors might be combined with existing passive or active 3-D viewingeyewear, thus optionally using the shuttering, narrow band filters (ornarrow band blockers) as additional means to multiplex views, enabling,for example, greater distance to enhance R-L contrast.

FIG. 44a shows the placement of projectors 4401 and 4402 above the pupilcenter base line 4403 with a vertical displacement of approximately 15mm. The inter-projector distance approximately equals the distance 4404between the pupils.

FIG. 44b shows an alternative approach, with the right-left projectors4411 and 4412 on each side of the eyes laterally displaced byapproximately 25 mm, thus adding a total of about 50 mm to theinter-projector distance.

FIG. 44c shows a top view of the light geometry 4430 when the projectoris placed on the side of the glasses. Projector 4431 projects beam 4432toward a splitting optics embedded in the lens. The optics reflects thelight from the projector outward toward the RR projection screen. Thereturning light is not reflected by the splitter, in this case becausethe light is polarized in one dimension, for example, vertically, andthe splitter is designed to reflect all the light with this state ofpolarization. One such polarizing splitter is made by a grid of nanowires, which can reflect light polarized in the direction of the wires(the direction induced current can flow). To such light the splitterlooks like a full metal reflecting surface. After reflection of thepolarized splitter 4433, beam 4434 is then turned 45 degrees by apolarizing element (such as a one-quarter wave plate) inserted in theprojection path. Upon retro-reflecting on the screen the returning waveis turned another 45 degrees so that the state of polarization of thereturning beam 4436 is now 90 degrees rotated or orthogonal to theoutgoing beam 4434, and the splitter is transparent. Note that theone-quarter wave plate can be laminated, or applied as coating, eitheron the outer lens surface or on the retro-reflecting screen surface.

FIG. 44d shows the projection beam path geometry 4440, which is similarto the path shown in FIG. 44c , except that the projector 4441 ismounted above the lens, as also depicted in FIG. 44b . Note that becausethe vertical displacement is shorter, the path traveled vertically bythe light coming from the projector is shorter, and because typicallythe projector's throw vertically is a smaller angle, the lens can bethinner and lighter.

FIG. 44e shows the projector mounted on the side, as in FIG. 44c , butnow the projector's light path is reflected at oblique angles by totalinternal reflection (TIR) inside the lens 4452 and reflecting off apolarized splitter 4453 toward the screen. The resulting, more oblique,angle of reflection allows the splitter to be angled more parallel tothe lens and allows the lens to be thinner.

FIG. 44f shows the same TIR reflection shown in FIG. 44e , but with theprojector 4461 on top of the lens 4462, yielding the thinnest andlightest lens in this series of options.

A Hybrid Retinal Projection System Scanning the Field of View Enablingan Advanced User Interface

In existing HMD eyewear, similar reflective and splitter optics are usedto project into the eye directly, creating a so-called retinalprojection. It is, in principle, possible to combine both functions inone set of eyewear, as depicted in FIG. 44g and FIG. 44h . Note that thetwo figures describe two functions of the same system at the left lens,seen from the top. A scanning laser “femto” projection engine (furtherdescribed in the discussion of FIGS. 35,36,38 and 39) creates a visiblelight pattern directly on the retina 4485, with visible light beams 4484(shown in FIG. 44h ), and said projection engine simultaneously projectsa matching pattern 4474 outward toward an RR screen, where it isreflected back as RR beam 4475, as shown in FIG. 44g . Returning back atthe lens, beam 4475 is reflected back toward the scanner 4471 anddetected by a sensor 4476. Crossed beam splitting devices 4473 and 4483,in the middle of the lens, redirect some of the primaries 4484 directlyinto the eye and others in the opposite direction. This arrangementenables projection of a highly visible image without requiring an RRscreen, providing see-through HMD functions and using minimal power. Atthe same time, outwardly directed beams scan the space ahead. Thescanning beams may be, for example, of invisible NIR wavelengths thatare reflected by a dichroic mirror, or Bragg-style narrow bandreflection on surface 4473. The outwardly scanning beam 4474 canstereoscopically probe for near objects, such as hands, as part of aninteractive gesture-based user interface system, or be pulsed to provide3-D ranging function. The key advantage of the arrangement is a light,wearable headset that provides an exactly eyesight-aligned sensingprojection combination. Such a function enables perfect 3-D image-objectalignment for augmented reality UI functions, for example, to help alignthe projected images with viewed realities (as depicted in FIG. 24) withsimple intuitive calibration options, as described elsewhere (asdepicted in FIG. 30).

One Size does not Fit all: Human Factor Adjustment Requirements inStereoscopic 3-D Imaging

Human stereovision (stereopsis) is a finely tuned brain function thatdevelops an integral part of our spatial awareness, that is, ourperception of the reality around us. Violation of this realityperception can cause significant discomfort, such as headaches, blurredvision, and nausea. Over-long periods of exposure to any such violationsof reality perception may seriously affect our natural vision abilities.The system and method disclosed herein includes the ability to detectand customize 3-D projection for individual vision requirements, thuspermitting a satisfactory and joint viewing experience on the samescreen for multiple viewers who may have significantly different humanfactor requirements.

Because no two eyes are precisely identical, the developing braindevelops maps that enable it to fuse the right and left view into asingle spatial 3-D perspective. This visual mapping process iscontinuous from early childhood. As we age or our eyes change, ouroptics are augmented or altered by glasses. The brain tries to readjustfor the distortions that are introduced. It is, therefore, notsurprising that no two eyes see the world exactly alike. Age has a verysignificant impact on vision, not only in accommodation range and depth,but in color perception and many other aspects. When creating anartificial reality, such as a stereo 3-D image, the more a system candetect and adjust for individual eye optics and vision characteristics,the fewer artifacts there are, and the more natural, the more “real,”and the more comfortable the experience is.

With existing means of creating the illusion of 3-D by presenting a“standard view,” left/right view disparities at incorrect focaldistances pose significant challenges to our vision system. Moreover,particularly in mobile systems, individual human factor adjustments arerequired within the 3-D view space to compensate for dynamicallychanging viewer-screen geometry.

Detecting and Adjusting for Interoccular Distance

Since the distance between eyes of individuals varies, detecting thisdistance and making the right adjustments for it is quite important toguarantee viewer stereo 3-D vision comfort and safety.

FIG. 30a shows a system 3000 according to one aspect of the system andmethod disclosed herein. By simple, natural hand-eye coordination,system 3000 can detect and adjust for individual interoccular distance.Dual head-mounted projectors (Eye Stalks) 3009 and 3010 locate RR screen3008 by, for example, detecting corner fiducials 3011 a-d. By meansdescribed elsewhere (see the description of FIG. 27) the systemdetermines the exact center of the RR surface (where diagonals cross).Because absolute screen dimensions (X and Y) are known, screen distanceis easily determined by measuring the angular extent of the rectangleformed by 3011 a-d in the projectors' field of view. The system thenprojects a 3-D point image (for example, a small ball) that is intendedto be perceived as floating approximately at one-half the Z distancefrom the screen.

Viewer 3001 is prompted to point the index finger 3005 of her right hand3013 exactly at the point where she sees the ball 3004 float. The dualRR shadows of the tip of the pointing finger is detected by projectors3009 and 3010, and because the projectors are at known spatiallocations, the exact positions on the screen of two fingertip shadows(not shown), referenced geometrically against the screen fiducials,allows the system to determine the exact spatial position of thefingertip 3004. Triangulation of actual finger position, matched withperception, suffices for the system 3000 to estimate the distance.

FIG. 30b (inset) shows an exemplary top view of this calibrationprocedure. When finger tip 3014 is held at one-half the distance Z tothe screen, and the ball floats in the same perceived one-half Z depth,then the disparity distance B between the ball's projected left andright images equals her inter-ocular distance A exactly. Other ballpositions left and right from the center and above and below the primarycenter of gaze—pointed at with a finger or pointing device—can furthermap the disparities required to faithfully represent 3-D, both in termsof horizontal and vertical disparity and the gradients of thesedisparities across an individual's view space.

Calibration to a Joint 3-D Reference to Ensure a Unified 3-D View(Shared Real-World Coordinates)

FIG. 30c shows two viewers A and B with projecting devices 3021 and 3022viewing RR screen 3020 in a procedure analogous to the one previouslydescribed in the discussion of FIG. 30b . Now one of the viewers holdsup a pointing device 3024 indicating where the small projected referenceobject is seen in 3-D. The projection devices 3021 and 3022 then finetune their exact disparities by adjusting the four images (one stereopair for each viewer), and the procedure can be repeated for variouscalibration positions. This procedure is enabled by both systems andboth players sharing the same screen fiducials and by at least onecommon view of a pointing device or real object in front of the screenas an additional spatial fiducial.

FIG. 30d shows how, instead of a pointing device, a small object, forexample, a part of a 3-D board game, may serve as an additional spatialfiducial reference to align two or more 3-D projections. In thisparticular example the board game has a horizontal retro-reflecting playprojection surface on which a miniature “steeple chase” jumping barrieris positioned somewhere. The surface has fiducial references such as,for example, the corners 3031 a, 3031 b, 3031 c and 3031 d, which allowboth projection devices to identify the exact center position andorientation of the play surface. Device 3032 of viewer A and device 3033of viewer B can clearly identify the real 3-D position and dimensions ofthe jumping barrier 3035, by referencing it against the RR screen belowit, in part by seeing the high RR contrast shadows (not shown) detectedwhere the object's features block each of the four projection beams (notshown) and (optionally) in part by motion parallax as the viewers' headsmove around the playing surface, using successive observations of thestationary object to determine its exact location. As with the playsurface, the game system may also have prior knowledge about the exactdimensions of the object. Having determined the accurate position of thebarrier 3303, in stable world coordinates, both players see the exactsame horse 3034 and rider just barely clear the barrier 3035 during thejump 3036 across the barrier. Having the two 3-D views unified into ashared and realistic 3-D experience makes the game interesting.

FIG. 30e shows the top view (as shown in FIG. 30b ) with finger 3045(outer circle) pointing at the perceived location of ball 3044 (innercircle). Note that there are two ball images 3046 and 3047 on the screen3041 seen by eyes 3043 and 3042, projected by Eye Stalk projectors 3049and 3048, respectively. After calibration as illustrated earlier in thedescription of FIG. 30a , the eyes see the real finger at the same 3-Dposition as the virtual ball. However, each projector, using feedbackfrom the photocell or camera in the Eye Stalk, detects shadow 3050, forthe left Eye Stalk, and shadow 3051, for the right Eye Stalk. Thesefinger shadows have a greater degree of disparity than the ball images;that is, there is a greater horizontal displacement between them. Byestimating the angular displacement alpha between the ball images 3047and the finger shadow 3050, and by knowing the distance to the screen Dand the position of the finger 3045, this procedure enables an accuratecalibration of the lateral offset distances 3052 and 3053 of the leftand right Eye Stalks 3048 and 3049 to each corresponding eye 3042 and3043.

If there are also Eye Stalk offsets in the two other directions (denotedas Z, toward the screen, parallel to the primary center of gaze 3055,and Y, for the vertical orthogonal to both Z and the baseline 3043direction X) they can be determined by analogous methods as described inthe previous discussion of FIG. 30 b.

In conclusion, by the methods of calibration described in the precedingsections, the exact positions of eyes, the Eye Stalks, and screenposition and orientation are determined. These geometries are then usedas the foundation for rendering a most realistic and comfortablestereoscopic 3-D perception, with a precisely rendered motion parallax,and with the right horizontal and vertical disparities for each objectin the 3-D space, as explained further in the next section.

Other Human Factors Adjustments: Viewer Head Turn (Adjustment Toward theGaze)

FIGS. 33a and b show the optical divergence in the Z-axis of an objectobserved outside the central view of a human. In these examples, thevertical (Z-axis) offset is roughly equal to the distance between theviewer and the screen, to exaggerate the effects. Though the human braincorrects for the optical disparity, a lack of such disparity maycontribute to the nausea some people get when watching CGI 3-D content,for example. This phenomenon (optical disparity) creates in some casesthe head-turn artifacts under discussion below.

FIG. 33a shows a viewer watching three object points 3301, 3302, and3303 in the fronto-parallel plane (for example, as seen in a projected2-D image). When the left and right eyes 3304 and 3305 fixate on themiddle object 3302, its image is centered in the fovea of each eye.Because it is closer, the retinal image of the left object 3301 hasgreater angular disparity than that of the right object 3303 (as shownin FIG. 33 a, 14° versus 12°). The total angular separation betweenpoints 3301 and 3303 is greater for the right eye because the eyeactually is closer to the objects. (As shown in FIG. 33a , the angularseparation is actually 10° more, 36° versus 26°.)

FIG. 33b shows a head rotated (approx 19°) toward the center of gaze,fixing on point 3312. In this example, the retinal angular disparitiesare significantly reduced. As shown in FIG. 33b , the angular disparityis reduced to 5°, half of what it was before the head rotation. Thetotal angular spread of the three-point image seen by the left eyeincreases to 28°, while for the right eye it decreases to 33°.

For virtual images, such as those created by stereoscopic 3-Dprojection, where the actual images are not at the perceived location,head rotation must be detected and compensated for to avoid distortionsoccurring as a result of such head movements. FIG. 34a and FIG. 34b showthe same geometric view, but now three points 3401, 3402, and 3403 areprojected in an imaginary plane 3409 at some distance Z in front of theactual screen surface 3408.

In FIG. 34a , again the initial head position is with the ocularbaseline parallel to the screen. The right eye sees 3401R, 3402R, and3403R, and the left eye sees 3401L, 3402L, and 3403L. The three pointsare perceived exactly as before.

However, as shown in FIG. 34b , to maintain these exact positions duringthe head rotation (the adjustment of the visual baseline toward thecenter of gaze) all the projected images must be adjusted. As shown inFIG. 34b , the shift of the left eye from position 3416 to the new, moreforward, position 3414 requires a rightward move by the points 3411L,3412L, and 3413L from their previous projection positions 3411L′,3412L′, and 3413L′.

Without such adjustment, the stationary images within a wide field ofview may wobble and geometrically distort during head rotations requiredfor adjusting the gaze in taking in a wide field of view. The experiencecan be disconcerting and interfere with motion stability during, forexample, interactive 3-D simulated training events.

Maintaining a fully natural 3-D vision experience during head motionsrequires instantaneous adjustment of horizontal and vertical disparity,horopter adjustments for off-center views, vertical horopter andtertiary fixation points (the effect of a greater vertical disparity forthe closer eye, known as the ipsilateral eye (see also FIG. 34c ,described below). All the effects of gaze changes are detectable by thesystem because it can determine the exact head position, thereforedetermine the viewer's baseline, frontoparallel plane, the planeparallel to the face, and orthogonal to the primary position of thegaze. Therefore the system can make adjustments to ensure thatartifact-free 3-D views are rendered for all vantage points by an autocalibration procedure analogous to the procedure described in theprevious discussions of FIG. 30a and FIG. 30 b.

Unique retinal structures and eyeball shapes are personal visionabnormalities that in many cases can be and need to be compensated forto maximize comfort when using 3-D images over long periods, forexample, in a work environment.

Following is a simple example of the improvement over standard 3-Dduring an activity such as, for example, watching Avatar in an I-maxtheatre: A flying object becomes visible in 3-D at the extreme top rightcorner of the screen. A viewer notices it, his eyes make yoked movementsrotating in the direction of the object to fixate on it, then (and onlythen) he naturally starts turning his head toward it, bringing theobject to the center of gaze position (turning the head from previouscenter-of-the-screen position) toward that upper right corner. The rightand left images should change in vertical disparity during thismovement. This is currently not the case in the theater.

FIG. 34c shows how the relative vertical disparity of two points P1 andP2, which, in this example, may be defined as relative verticaldisparity=(β1L-β2L)−(β1R-β2R) (see Ian P. Howard, Brian J. Rogers:Binocular Vision and Stereopsis, p. 282), depends on how far the pointsare angularly removed from the primary center of gaze (the point offixation when looking straight ahead). Close objects at the outer rangesof binocular view have the greatest vertical disparity. For realisticmobility training, in sports for example, the vertical disparity is animportant effect to get right so a viewer can duck a projectile comingfrom the side. As with horizontal disparity discussed in the descriptionof FIG. 34b , vertical disparity is reduced by the viewer turning towardthe object. This head rotation, bringing the object toward the midsagital plane, reduces both horizontal and vertical disparity.

Embossed Spherical Retro-Reflection Structures

FIG. 29 depicts an example of a dual-radius spherical retro-reflector.Light ray 2905 impinges on the front spherical surface of radius R1 withan angle of incidence i of 30 degrees. It is then refracted andreflected at a spherical back surface 2902 with a radius R2 greater thanR1 but with the same center of curvature 2907 as the front surface 2901.Choosing the refraction index N of the lens material correctly enablesthe refracted beam to be directed toward the bull's eye 2904, which isthe intersect with the spherical back surface of a line through thespherical center 2907 and parallel to the incoming ray 2905. Reflectingon a reflective coating 2903, the ray symmetrically continues back tothe front and exits exactly parallel but in the opposite direction ofray 2906. In this example the ratio R2/R1 is 1.4142 (square root of 2),and achieving perfect retro-reflection at an angle of incidence of 30degrees requires an index of refraction of 1.65. The advantage of thisstructure is that the top and bottom surfaces 2901 and 2902 can bemolded (embossed) at the same time from a sheet of plastic material,after which the back surface can be coated with a metallic reflector(2903) and optionally with additional structural filler and/or adhesivelayer 2808. The light incident at other angles is reflected in thedesired “doughnut cone” distribution pattern. The advantage of thisdesign over the prior art of a surface coating with highly refractivemicrospheres is twofold:

1) This design is moldable as one simple solid form structure, due tothe extra distance between the back and front surfaces. By contrast,spherical reflectors with single-radius or dual-shell designs requireadditional coatings, adhesives, assembly, and many more manufacturingsteps.

2) Due to the larger back radius, this design requires a lowerrefraction angle and therefore can use lower index materials, forexample, 1.65 as compared to 1.9 or higher, which materials are moreeasily sourced and less expensive.

Personal Projection Prompter Mobile Teleprompting Device

FIG. 31a and FIG. 31b show a system that can, by using retro-reflectivesurfaces to stealthily display text and images, assist a presenter, anactor, or a conference attendee in a manner invisible to the audience.

In FIG. 31a , the projected text is invisible to the audience, becausethey are not looking at it from the presenter's exact angle, and it istherefore outside the RR view cone. Due to the high-gain nature of theRR surface, the light budget is low enough to allow a mobile personalprojection device no bigger then a clip-on microphone to projectreadable text at more than 100 feet away in a low-ambient-lightenvironment such as back walls of a theater.

The mobile teleprompting device can receive wirelessly or display fromlocal memory. RR surfaces can completely surround the audience, (walls3101 and 3104, stage floor 3105, or even a ceiling), and text may beplaced wherever the presenter is viewing at that time. Text fields 3103a and 3103 b wrap around obstructions in the field of view of thepresenter. The presenter can walk around, see his prompts, yet maintaineyeball contact with the audience. Thus a “virtual teleprompter” systemis created, which system may be miniaturized to no more than a single 3mm Eye Stalk attached to or integrated with the existing wirelessmicrophone. The projected text can be big and easy to read at acomfortable (infinite) focal distance. The text may be “anchored” in afixed position to the RR surface or move with head motion as required.Alternatively, after the presenter changes position or gaze, new promptsmay start appearing within the new gaze range.

A beam skip features is used to make the projection and the device'slight emissions invisible. The device only emits light, such as brightletters on dark background, as long as the retro-reflective projectionsurface is detected. This simplest detection mechanism is a fastphotocell mounted next to the scan mirror (or in the same path using apolarized beam splitting mechanism). Whenever something is between thepresenter and the screen, the projection skips to the next available(unobstructed) screen position. For example, a member of the audience3102 stands up into the path of the projection. The beam is scanning atleast 18 kHz scan rate. Within a millisecond, the projectors, scanningthe next line, skip over the area where the obstruction is detected byturning off the visible light. Neither cameras nor the human eyeperceive a scanning beam emanating from the presenter's headset.Projecting in both directions permits up to 36,000 lines per second tobe drawn. For example, a narrow 300-line text banner can be updated andmotion-stabilized at a speed equivalent to 120 frames per second). Caremust be taken to minimize or eliminate stray light leaking from theemitter.

Additional mechanisms can prevent the presenter prompter frominterfering with a video recording camera. Such mechanism may include,for example, using synchronization for the prompter projector to emitbetween camera shutter exposure intervals, or using narrow band blockingfilters, such as, for example, using a narrow band laser in theprojector. Since this band can be less than 1 nm wide, it can befiltered out without causing serious color artifacts assuming broadspectrum studio lighting. In other cases, laser light is naturallyhighly polarized, while, natural, performance venue, or studio lightingis not. Thus a polarizing filter can be added to the cameras if requiredto filter out the laser light and prevent it from being seen in closeups of the performer.

A UN-Style Multi-Lingual Conference System with Live MulticastTranslation Subtitling

FIG. 31b shows an additional novel application for the “teleprompter”head mounted projection system: providing real time translated voice totext. The walls of a conference room 3150 are covered with RR surfaces3150 and 3152 that allow personal devices to display (for the user'seyes only) a translation (subtitles) of what another party in theconference is saying, translated into the required language. Forexample, when Japanese party 3154 (A) says “Ohaio Gozaimas,” hismicrophone picks up the audio feed and sends it to a networkedtranslation system that translates it to “good morning,” which is thenwirelessly relayed as text to the headgear 3157 of the other party 3155(B). Said headgear projects this text on the wall behind party 3154 as akind of “text balloon” that is clearly readable to party 3155 from wherehe is standing. Analogously, when party 3155 answers in English, hisspeech is translated back to Japanese and converted to text for party3154. Should either party need help with their presentations, they canbe prompted or notified of important personal (private) messages on anyRR surface that happens to be in the party's field of view at thatmoment. Such a system might also be an extension of the stage promptingsystem, where the surface viewable primarily by the performers on, forexample, the stage side walls could have additional RR informationspaces, such as on the table 3158.

A further use of such a facility may be for assisting conversations forthe hearing impaired.

Open, Flexible Work Environment, Shared Telecommuter Facilities, QuietRooms, Libraries and Class Rooms, War Rooms, Control Rooms

The system and method disclosed herein may be used, for example by agroup of workers sharing a wide open area. They share large worksurfaces, both tables and walls (and windows), where they can projectall sort of personal views. These views overlap, and through headsetstheir video and audio is multiplexed. Workers entering and leavingcontinue seamlessly to use their personal projectors, such as, forexample, new 3-D system-enhanced Blackberries) or other system-enhancedpersonal mobile devices that they are already using outside the offices.Now, however, they have full access to all the extra local facilities(fast server access, secure data access, dedicated telepresence highgrade QoS networks). In these facilities each person has almostunlimited virtual desk space and virtual personal display space. Mostimportantly, subgroups can spontaneously start sharing complex data,such as 3-D graphs and images, and collaborate as tightly meshed teamsusing ad hoc telepresence connections, naturally interacting with eachother and with remote teams, with minimal disruption for those notinvolved. Every surface is used multiple times. This approach reducesthe need for dedicated offices, conference rooms, or special facilities,and high-grade, high-cost, HQ-style facilities are optimally utilized.Most importantly, pervasive telepresence and collaboration supportfosters both team and personal efficiency, breaks down corporate wallsand so-called “silos,” and allows for a highly mobile, versatileworkforce deployment.

“Invisible” Embedded Retro-Reflective Fiducials

FIG. 32a shows a retro reflective layer 3210 that is embedded on anotherdisplay surface, which may be a specular (3211), a retro-reflective(3209) or a diffuse (3204) reflecting surface. The retro-reflectivelayer 3210 in this case is only retro-reflecting certain wavelengths,while it is transparent to others. For example, IR ray 3201 is shown toretro-reflect as beam 3202, whereas visible-light beam 3205 is reflectedspecularly as ray 3206. Thus for visible light the surface acts in thiscase as a mirror. Alternatively, the visible light beam 3207 is shown toretro-reflect as a slightly diffused (“donut cone”) beam 3208, or,alternatively, projection beam 3203 is projecting a regular image on adiffuse back surface 3204. It should be obvious that combinations ofvarious types of reflective surfaces can be composed in this matter. Forexample, retro-reflective fiducials might be embedded in a projectionsurface, in this aspect to help guide the scan and to ensure autoalignment and safety features. A wavelength-selective retro-reflectivelayer may be constructed by, for example, applying a Bragg-type narrowband reflecting layer to a corner cube structure embossed into atransparent base material. If the material below and above the Bragreflector is optically matched, this approach should minimize unwantedbroadband TIR back reflections in the RR structure's back surface, andonly light in the narrow waveband of the Bragg grating is retroreflected by layer 3210.

FIG. 32b shows an example of a projection screen 3230, said screen has amain projection surface 3232 and is framed by a retro reflective borderstructure 3231, constructed as described previously.

FIG. 32c shows a cross section of the frame with the narrow-bandretro-reflecting structure 3233 as an additional layer on top of thescreen material 3235. Optionally, an additional protective coating 3234protects both the screen surface and the RR border fiducial 3233. Notethat images can be projected on the entire screen 3230 including the RRborder 3231.

A Highly Integrated, Compact Femto Projection Multibeam Laser Source

FIG. 35a is reproduced from Sony U.S. Pat. No. 6,956,322 B2. The secondembodiment (FIG. 11 in the patent, renumbered here for clarity) teaches“a light emitting device 3500 has the first light emitting element 3501capable of emitting light in the band on the order of 400 nm and thesecond light emitting element 3502 having the lasing portion 3503capable of emitting light in the band on the order of 500 nm and thelasing portion 3504 capable of emitting light in the band on the orderof 700 nm.”

FIG. 35b , also taken from Sony U.S. Pat. No. 6,956,322 B2, has beenmodified to show light emitting device 3500, which is constructed bystacking two different semiconductor material layers (for example, GaAsand GaN) with one semiconductor layer containing one lasing structureand the second layer containing two lasing structures. Each lasingstructure emits light at a different wavelength and can be modulatedseparately. The patent teaches how a stacked light emitting deviceemitting light of “three primary colors red (R), green (G) and blue (B)”from three cleaved mirrored facets 3514, 3515 and 3516 can be used as a“light source of not only the optical disk drive but also full-colordisplays.” This design was driven by the need to simplify and costreduce the light source assembly of a BluRay™ optical disk, sincecombining the light sources in a litho-graphically exact replicatedgeometry defined by the triangle 3515 significantly reduces thecomplexity and cost of the optical assembly.

Improvement on the Above for Constructing a Miniaturized Light Enginefor an Ultra Compact Femto Laser Projector or 3-D Image Generating EyeStalks

The design consists of five or more lasing sources mounted on two ormore dissimilar layers of semiconductor material. As per the same patentcited above, the advantage of using a stack is that currentsemiconductor technologies require one type of materials for the longwavelengths, for example, IR, red, orange, and yellow, and anotherchoice of materials for shorter wavelengths such as green and blue.

The design's particular choice of wavelengths is, among others, drivenby the following criteria:

1) Luminous Efficiency: Luminous efficiency may be summarized as lumensout per electrical watt in. The goal is the brightest perceived imagerealized with the least amount of electrical power. Generally there is atradeoff between luminous efficacy (LM/mW light power) versus so-called“wall plug efficiency” (WPE) of a laser source (mW light out per mWelectrical power in, typically specified as a percentage ranging from 5percent to 45 percent).

2) Color Range: The projector must be capable of a wide range ofunsaturated colors (gamut)

3) Cost Reduction: Reducing cost by minimizing complexity of the opticalassembly. In optics it means typically fewest alignments.

A scanning projector can scan multiple collimated beams simultaneouslyoff the same scan mirror (or other beam-steering devices, such aselectro optical modulators). Therefore, the number of primaries can begreater than three, without unduly increasing the complexity or the costof the design. This principle is analogous to today's low-cost inkjetprinters, which all use more than three colors to realize thebest-gamut, highest-color, accuracy with the least ink. By using asingle, light-emitting device capable of as many as six wavelengths inthe range of 400 nm to 900 nm, a highly efficient and compact design canbe realized. Because the high gain of the RR surface significantlyreduces the projection power requirements, the individual lasingstructures require very modest current and power densities, andtherefore they can be packed together at a less than 100 micron pitch. A1 mm by 1 mm chip can easily accommodate a plurality of such structuresat no incremental cost. A light-emitting device stacking two layers withthree lasing structures each can emit six different wavelengthsindividually modulated with picosecond precision.

Five Visible and One Invisible Primary

Laser and LED light sources are available in a broad “palette” ofwavelengths, such as, for example, 440 nm (deep blue), 480 nm (blue),500 nm (blue green) 525 nm (deep green), 575 nm (bright yellow), 590 nm(orange), 650 nm (red), 680 nm (deep red), 850 nm (NIR). The latterinvisible NIR “primary” is optional, but the advantage of adding it isthat it serves as a tracer bullet, providing traceability of the scanpattern regardless of image intensity. It also serves as a means ofidentifying and tracking the location of the screen, any fiducials, andthe shadows of hands, fingers and objects.

Why Use More than Three Visible Primaries?

There are several reasons for using more than three visible primaries.First is the almost negligible incremental cost. Given that the aspectratio of a laser stripe (in an edge emitter) is approximately 1 mm inlength, making a laser diode wider than one stripe is necessary formechanical reasons, thus the extra 2 stripes per layer do notsignificantly add to the structure's size.

The second reason is for maximizing efficiency. FIG. 41a shows anormalized eye cone response curve (source: Wikipedia). Human eyebrightness perception (luminous efficacy of perceiving bright imagesthrough photropic, that is, daytime, vision) peaks around 555 nm at 683lm per watt, but the three types of retinal cones have sensitivity peaksat three distinct wavelengths of around 565, 535 and 440 nm,respectively, for the L, M and S type cones). The response maxima of theL and M cones are quite close to each other and their broad sensitivitycurves overlap greatly. Primaries in the 535 nm to 565 nm range actuallyappear bright yellow since they stimulate both L and M cones. Perceptionof colors such as deep red (toward 650 nm) and deep green (toward 500nm) require a strong differential in L and M cone response.

The central foveas (normally center of focus) are almost two-thirds L(red favoring) and one-third M (green favoring cones). The “blue” conesare identified by the peak of their light response curve at about 445nm. They are unique among the cones in that they constitute only about 2percent of the total number and are found outside the fovea centralis,where the green and red cones are concentrated. Although they are muchmore light sensitive than the green and red cones, it is not enough toovercome their disadvantage in numbers. However, the blue sensitivity ofhuman final visual perception is comparable to that of red and green,suggesting that there is a somewhat selective “blue amplifier” somewherein the visual processing in the brain. (Source: Human Physiology FromCells to System, Lauralee Sherwood)

The above implies it would not be advisable to put so much spatialfrequency blue at the center of focus (at the image position that isaligned with the fovea centralis) because it is not seen anyway. Maximumspatial contrast would be most efficiently achieved with the yellowrange red and green primaries; that is, green, yellow, or red modulationaccuracy matters in the place vision fixates. Knowing where human visionfixates is valuable. A good guess would be vergence (3-D disparity) andany moving objects. A system such as described in this disclosure might,for example, maximize red and green high-speed spatial detail contrastgeneration for moving objects on which the human vision automaticallyfixates.

To maximize brightness and luminous efficacy, it would seem advantageousto use primaries that closely match these individual peaks in the conesthat drive photropic vision, but with only three monochrome primariesmatching the conal peaks, the color gamut would be severely truncated onthe red side. FIG. 41b shows the CIE 1931 chromaticity diagram and therelatively narrow triangular subset 4110 formed by the three primaries4111, 4112, and 4113 with wavelengths matching the conal sensitivitymaxima of S, M, and L cones (440 nm, 535 nm, and 565 nm respectively).Clearly colors that fall in areas 4114 and 4115 cannot be rendered by asuch a system. Adding primaries overcomes this limitation and achieves abest-of-all-worlds situation. Where and when required, just enough ofthe deeper color, that is, longer wavelength RED, is added to the mix torender the correct hue. The more efficient primaries carry the bulk ofthe imaging load in terms of delivering brightness and creatingcontrast. Note that while the above mechanism for achieving wide-gamutrendering efficiency would work especially well for lasers that tend tonaturally emit narrow unsaturated monochrome colors, it also applieswhen using spectrally more diverse but narrow primaries such as LEDs orhybrid LED laser devices (resonant cavity LEDs, quantum dot lasers,etc).

Another reason for using more than three visible primaries lies in theefficiency trade-offs between device luminous efficiency vs. wall-plugefficiency. At certain wavelengths of high luminous efficacy, deviceswith acceptable wall plug efficiency (mW light out for mW electricalpower in) are not yet available. For example, direct green laser diodesaround 525 nm made by Soraa and Osram still have only 3-5 percent WPE.It may, in such cases, be preferable to use, for example, shorterwavelength “bluish” greens because the increase in laser efficiency(WPE: mW out per mW in) more than offsets the concomitant decrease inluminous efficacy (lm per mW out). In general, having primary alternatesavailable in the spectrum results in having more choices for renderingcolors of the desired hue and brightness, and it can only helpefficiency. The operational power savings from having one or morealternate primaries would depend on something akin to “color demandstochastics,” that is, population density of chromaticity requirementsas distributed across the gamut. Optimality would be testedstatistically by summing the product of probability of a certaincolor—P(Ci)—with the efficiency of generating that—E(Ci)—with a givenchoice of primaries—Ci=f(p1, p2, p3, p4, . . . pn)—where typically theclosest subset of three (in some cases two) primaries would be the mostefficient. It is clear that if deep blue seas and fields of very redroses were seldom in the images, the savings from being able to use moreefficient alternate primaries most of the time would be significant.

An additional reason for using more than three visible primaries is tomitigate speckle image degradation, which can be a major problem inprojection. It is an inherent problem when using narrow-spectrum,coherent light sources and small aperture optics. Speckle must bemitigated in several ways. Generally, the less source coherency thebetter; therefore, when possible, it is preferable to increase thebandwidth of primaries, to shift phases, and to increase the number oflaser resonance modes. Low power semiconductor laser diodes with narrowstripe structures typically individually produce very coherent light ofa single frequency, in single mode. After reflecting off the screen, thelight of each such a structure tends toward self-interference, creatingpeaks and valleys of brightness, known as “speckle.” Having morestructures that are not coherent and not of the exact same frequencyhelps to mitigate speckle. A multiplicity of N uncorrelated lasingstructures yields a more homogenous image (as perceived by the eye). AsN increases for any given time period and retinal position, the sum of Npatterns becomes more homogenous (with greater N, speckle “averagesout”).

Additionally, because speckle is primarily perceivable as a fixedpattern noise, it can be mitigated further by making a slight “scramble”of the pixel positions, for example, by introducing a third-order,somewhat random variance beyond the required x and y periodicity in thescanner's motion. A temporally and spatially scrambled pixel map and achanging scan beam pattern result in the viewer experiencing atime-integrated overlay of multiple speckle patterns, thus reducing theperceptibility of speckle. Note that the output of a lasing structurenaturally tends to change slightly in wavelength and phase during itsrise and fall. Hence, lighting an image edge, for example, during aleft-to-right scan, creates a different speckle pattern than whenimaging in the other direction (the return, the second phase of thehorizontal scan), so overwriting a left-to-right scan detail withright-to-left scan in the next frame also helps reduce speckle. Thisoverwriting can be done by adding a slight offset (by 1 line width, or afraction of a line width) in the horizontal scan pixel map, frame toframe. Image “pixel positions” are somewhat arbitrary and can beaccurately interpolated by high performance GPU capable of rendering anyresolution raster position ad hoc, and within less than one frame delay.The resulting image (as perceived by the eye) is not degraded or blurredby adding this “pseudo random” element to the scan pattern.

A Projection System Using Five Primaries: Efficiency and OtherAdvantages

FIG. 40 shows a CIE 1931 2° standard observer chromaticity diagram. Aset of standard R (642 nm), G (532 nm) and B (442 nm) render a gamut aswide as 150 percent of the NTSC gamut, enabled by the spectral purity ofthe RGB laser diode sources (the NTSC gamut is based on CRT phosphors,which are less spectrally pure). However, the luminous efficacy (lm peroptical watt) of the 642 nm red primary is only 109 lm per watt, andthat of the 442 nm blue is a minimal 18 lm/watt. Therefore, colorscontaining significant amounts of red or blue, such as bright whites andthe less saturated colors in the center of SCIE 1931 chart, require alot of optical power.

The situation changes drastically with two extra primaries, such as Y(yellow) at 565 nm and X (blue green) at 495 nm. These two primarieshave much greater luminous efficacies, 557 lm/watt and 117 lm/watt,respectively. Note that, for example, those colors that fall in region Vcan now be entirely rendered by using primaries X and Y instead of R andB, with a gain 5× in efficacy. Substituting for deep red (longwavelength) and deep blue (short wavelength) with less extreme colorscan save up to 80 percent of light. Also note that adding the X (bluegreen) significantly extends the gamut. Furthermore, for most of thecolor space, any one color can be rendered from several differentcombinations of 3, 4 or 5 primaries (known as metamers, colors withdifferent spectral composition perceived as identical). Somecombinations might be more efficient than others, but in some casesmixing in a fourth or fifth primary might help improve image fidelity.Inefficient or power-limited sources can be substituted for withproximate primaries. For example, direct green laser diodes still havethe relatively low wall plug efficiency (WPE: 3-5 percent). X (forexample, a more plug-efficient bluish-green GaN laser diodes) and Y(bright yellow) can wholly or partially substitute for G in five areasof the gamut (all but V and VI). In all cases more spectral diversity—asnoted before—minimizes speckle and other artifacts. For example, skincolors are generally rendered more accurately across visual age groupswith a four-color system).

Note that the rendering palette can be instantaneously adjusted,switching back and forth between power saving mode and extreme colormode (wide-gamut mode), triggering real time on content requirements. Aspecial “paper white” mode renders bright white-on-black textparticularly efficiently. Whites are renderable from several differentprimary combinations, ranging from highest efficiency, to highestdetail, highest brightness, lowest speckle, ease of accommodation, andfocus (using a narrowest bandwidth to render white minimizes chromaticaberrations), etc., and any such tradeoffs can be made within the images(partial fields) based on the GPU advance knowledge in the renderingpipeline, and/or based on detecting the image requirements themselvesand optionally from feedback from the rendered images. Since specklebecomes more visible in objects such as, for example, homogeneous brightobjects under fixation in the fovea, ex ante image analysis (by GPUsoftware in the rendering pipeline, for example) and determination ofthe primary focus of gaze (for example, by detecting head movementsdenoting fixation on the object in the view plane) determine if specklemitigation measures such as spectral diversification are warranted.

Guiding Light

Among many, there are two possible simple approaches for guiding light:

First, the light-emitting element may be integrated close to or with thescanning optics. In case of the Eye Stalk dual-projector design, thisapproach requires two separate optical sources, each with its ownpackaging and supporting electronics.

Second, all the required light sources may be co-located in one package,possibly integrated into a single semiconductor device with 6-10 (ormore) modulated beam outputs, and two combined beams are guided towardseparate scanning mirrors via an optical waveguide, such as a fiber.

The advantage of the first approach is that all the optical-mechanicalassembly is contained within the Eye Stalk, and only electricalsignaling and power needs to be provided externally by, for example, awire as with ear buds from a host device.

The advantage of the second the approach is that all of the power andheat (losses of laser diodes) is kept away from the Eye Stalk, reducingthe complexity and possibly the size of the Eye Stalk, adding to thecomfort of the wearer. However, alignment and coupling of the fiber onboth ends is required. Such alignment may be achieved, for example, byusing conventional fiber optic coupling technologies, such as by Vgroove or ferrule insertion.

FIG. 36a shows the multi-primary engine 3600, similar to the onedescribed in the discussions of FIG. 35a and FIG. 35b . The laser diodestack 3601 is mounted on a D-shaped heat sink 3602 that can be fit in aTO package with a ferrule type of optical combiner lens element 3603that slips over the assembly. The output is a single collimated beam(not shown) combining all primaries, or, as shown in FIG. 36b , awaveguide or ray optics type of combiner-coupling element 3613 thatcouples light output of the multi primary diode stack 3611 into the core3615 of a fiber like waveguide 3614.

FIGS. 38a, 38b and 38c show examples of using a refractive collimationof a six-tripe diode stack, as described earlier in the descriptions ofFIG. 36a and FIG. 36b (for example, five visible primaries and IR).

FIG. 38a shows a side view of the six-diode system 3800, comprised of atwo-layer stack 3801, with each layer containing three laser stripes(only lasing structures 3802 a-3802 d are shown in the side view). Afast cylindrical lens 3803 collimates the fast axis, followed by a slowcylindrical lens 3804 collimating the slow axis. The resulting sixcollimated beams 3807 converge into a single spot 3806, for example,within a scanning mirror 3805.

FIG. 38b shows the top view of the same configuration. Lasing structures3812 a, 3812 b, and 3812 c of the top layer of the six-laser diode stack3811 are shown. A fast cylindrical lens 3813 collimates the fast axis,followed by a slow cylindrical lens 3814 collimating the slow axis. Theresulting six collimated beams converge into a single spot 3816, forexample, within a scanning mirror 3815.

FIG. 38c shows the top view of a dual system 3821 with two sets of sixprimaries (a total of 12), where the collimated outputs of each set arecoupled into the cores 3822L and 3822R of flexible waveguides 3823L and3823R. This arrangement allows all the light sources to be containedwithin a single semiconductor-optical structure away from the rest ofthe projection scanning optics, minimizing the physical dimensions ofthe Eye Stalks

FIG. 39a shows the top view of a dual system 3900 with another possiblebeam combiner embodiment using an optical waveguide structure with aplurality of refractive index waveguide channels in a merging combinerpattern. The system consists of a diode stack 3901, a horizontalcombiner 3902, a vertical combiner 3903, and a ferrule or v-groove typeof fiber alignment structure 3904 leading to two fiber waveguides 3905Rand 3905L.

FIG. 39b shows the side view of the same system 3900. Note that becausethe dimensions of all the components are precisely controlled andaligned in the same directions, assembly is simplified.

Scrambling Light in a Slightly Irregular Array of Retro-Reflective CubicCorners

There are basically two types of standard (perfect) cubic retroreflectors. In the first type (see the discussion of FIG. 46g ), thethree sides of a cube are cut diagonally at a 45 degree angle, so thatthe base of the trihedral pyramid is shaped as an equilateral triangle(three equal sides with 60 degree angles). When all cube corners are inoriented the same direction these equilateral triangles tile into aregular polygon.

In the second type (see the discussion of FIG. 46a ) the cube sides aresquare and the tips of the three sides extend upward from the plane sothe base, as seen from an angle, looks like a hexagon, or as an array itlooks like a honeycomb. Note that the base is actually not flat, butjagged.

As noted earlier, in some cases is desirable to open up the reflectioncone somewhat so the reflected beams from a near-the-eye-projector aremore easily seen. The resulting pattern is described as a doughnut orhollow cone reflection. For a cube corner type retro-reflector thischange can be made by varying by a small angle Alpha away from perfectorthogonality. For example, adding 1 degree would make the angle betweenat least one set of the three reflecting planes 91 degrees instead of 90degrees. Thus the two types of RR cube corners can be modified and thebase patterns also become slightly irregular. For example, if a smallangle is added to only one of the planes in first type, the base patterncannot be a perfectly equal sided triangle, so tiling the pattern into aregular polygon becomes more complicated. However there is a significantadditional benefit from adding such an irregularity to the RR array: theirregularity can be used to scramble the coherency of incoming waves andreduce the speckle that might otherwise be seen by the observer of theimage.

A perfect RR corner cube of type 2 has three planes to every facet. FIG.46a shows planes 1, 2, and 3. Incoming light can land on any one of thethree planes. From there it can travel to one of two adjacent planes. Intotal, each facet has six sub-apertures, shown as 1 a, 1 b, 2 a, 2 b, 3a and 3 b in FIG. 46b . FIG. 46d shows the six different optical pathsafter the first reflection on one of the cube corner planes. Asdescribed previously, any one of the planes can be rotated slightly by asmall rotation angle a, as shown in FIG. 46c , to modify the lightreflection angle from pure retro reflectivity to create a wider, morediffuse doughnut reflection pattern. FIG. 46e shows six equivalent pathsslightly deviating from the pure retro reflecting paths shown in FIG.46d . In a perfect retro reflector as shown in FIG. 46d , a coherentwave front going through three successive reflections travels the samedistance and experiences the same phase shift. Due to symmetry, thelengths of the paths are identical, both in type 1 and type 2. When theretro-reflecting structure is altered, as it is in the altered irregularstructures shown in FIG. 46c and FIG. 46e , this is no longer the case.

A variation of one or more of these factors in the RR-faceted array canconsist of the following:

a) Angular offset (varying the size of alpha; the small angle added).

b) Varying which and how many (1, 2 or 3) of the RR cube angles are not90 degrees.

c) Orienting the cube corners' main axis away from orthogonality withthe main plane describing the array. Offsetting or slightly misaligningthe angles of these axes also increases the RR surface acceptance anglebeyond the acceptance angle of the individual facets, thus softening theenergy peak as described in prior art (see U.S. Pat. No. 3,817,596). Butanother significant advantage is this approach is that it scrambles theparts of the incoming wave front. The wave front hits facets at manydifferent angles, and since the path traveled varies (in the high Nmaterial) this approach results in a quasi-random phase shift of thewave front, helping to reduce speckling artifacts.

d) Various other slight variations in the shape of the sides of the cubecorners, either of type 1 or type 2, are possible. The sides can bedivided into facets themselves, with a slight angle between two facets.For example, in a full cube corner, the three squares of planes 1, 2,and 3 can be divided in two triangular parts (by dividing one of thesquares along the diagonal, for example) with slightly differentorientation (again, small offset α of perhaps 1 degree).

FIG. 47a shows a tiled configuration of type 2 retro-reflecting facetsarranged in a plane, with each facet slightly modified by one or more ofthe methods described previously in the description of FIG. 46, above.In the example shown, the shaded areas with a slight orientationrotation α are out of perfect orthogonality with the adjacent planes.

FIG. 47b shows a tiled configuration of type 1 retro-reflecting facetsarranged in a plane, with each facet slightly modified by one or more ofthe methods described previously in the description of FIG. 46, above.In this example, the shaded areas with a slight orientation rotation αare out of perfect orthogonality with the adjacent planes.

Use of a Square Fiber as a Multimode Scrambler to Reduce Speckle

A square fiber works as a mode scrambler, reducing spatial coherency oflaser diode light coupled into the fiber. The shape of the core of thefiber guiding the light from the diode stack (see the previousdiscussion of FIG. 38c for more details) therefore can be used to modifythe guided light to reduce speckle.

Virtual Hands: User Interface Facilitating Hand-Eye Coordination inAugmented Reality

It is desirable to be able to naturally interact with virtual objectsprojected within our reach. Hands are the most natural way to do so. Ourhands have superbly evolved 3-D motion and manipulation abilities thatcannot be bestowed on a mouse or other novel pointing devices. Opposingthumbs come with advanced hand-eye coordination, allowing us to thread aneedle and catch a ball. Our binocular depth vision is acute in therange our hands can reach. We have strong motor feedback to the relativeposition and motion of our fingers, but only by seeing our fingers withboth eyes do we get good spatial clues in terms of where our fingers arewith respect to objects not yet touched.

It is easy to fool the brain to see our hands transposed in an image ina position that is not their actual position. This transposition can beexploited to solve the shadow and occlusion problems that occur when outreal hands move between our eyes and the projected image. A real handoccludes a virtual ball when the hand is reaching behind it, when infact the ball should occlude the hand.

The solution is to observe the user's hands in a position outside thedirect view cone in which the object appears and project a set of“virtual hands” in stereo 3-D and allow these hands to manipulate theball. The virtual hands can be approximate or exact copies of the actualhands as observed by a camera or scanning device, or they might be morecartoonlike images that motorically mimic the actions of the real hands.The virtual hands appear to the viewer to move and touch exactly likethe viewer's real hands. The virtual hands are perceived as an extensionof the body, turning the hands into perfectly natural pointing andmanipulation devices. One example of how this phenomenon might beinstantiated is described below, in the discussions of FIG. 45a throughFIG. 45 c.

FIG. 45a shows a user 4500 who sees the 3-D image of a ball 4503floating approximately half way toward the screen 4502. His hands 4504and 4505 are being scanned by a pair of projectors 4506 and 4507,casting strong IR shadows 4510 on the lower section 4502 a of the screen4502, while images of the ball 4503 and the virtual hands 4508 and 4509are projected in the upper part 4502 b of the screen 4502.

FIG. 45b shows the front view of the screen 4512 with the upper section4512 b with virtual hands 4518 and 4519 and the lower section 4512 awith IR shadows 4514 and 4515 of the hands, invisible to the viewer butclearly seen by the photocell feedback system of the projectors (notshown), as described throughout herein. The viewer sees the hands in thesame pose, with the same scale and precise finger motions, and thus hasa completely natural visual feedback. The virtual fingers can touch theball from behind, while the ball is moving forward, without interferingwith the correct occlusion. (Parts of the fingers and hand are behindthe ball and each eye sees the correct relative occlusion, whichocclusions are quite different due to the large disparities at closerange).

FIG. 45c shows a top view of the viewer's view cone projected on theupper section of the screen 4522. The left and right images 4521 a and4521 b of the ball are projected by projectors 4526 and 4527respectively onto the screen 4522. (The images are drawn with anincorrect front perspective rather than a top perspective for clarity).The viewer's eyes 4524 and 4525 fixate on these images, and hisbinocular vision fuses them into one 3-D image 4523. Similarly, theprojectors render 3-D images of both the left and right hands (shownhere only as index fingers 4528 and 4529). Note that the real hands inthe line of projection would have created multiple shadows and wouldhave occluded much of the ball. This problem is entirely eliminatedwhile fully naturalistic looking “virtual hands” are correctly insertedin the 3-D view.

FIG. 45d shows how a real hand 4530 catches a virtual ball 4531. Thereal hand's motions during the catch are transposed to the virtual hand4532 into the projected image. The graphics rendering system ensuresthat the virtual ball 4532 correctly occludes the catching hand in thecomposite image 4533. Note that while not shown in stereoscopic 3-D,there are two images, one for each eye, each image rendered from aslightly different perspective. In each view different parts of the handand the ball are seen and occluded.

It is clear that many modifications and variations of this embodimentmay be made by one skilled in the art without departing from the spiritof the novel art of this disclosure.

For example, in some cases the system may project a user-viewable,computer-generated or -fed image, wherein a head-mounted projector isused to project an image onto a retro-reflective surface, so only theviewer can see the image. The projector is connected to a computer thatcontains software to create virtual 2-D and or 3-D images for viewing bythe user. Further, one projector each may be mounted on either side ofthe user's head, and, by choosing a retro angle of less than about 10degrees, each eye can only see the image of one of the projectors at agiven distance up to 1 meter from the retro-reflective screen. The retroangle used may be reduced with larger viewing distance desired. Theseprojectors may use lasers to avoid the need for focusing, and theprojector may use highly collimated LED light sources to avoid the needfor focusing. Also, at least one camera may be mounted near a projectoron the user's head and may be used to adjust the image or used toobserve user interaction with the projected image. In addition, a beamand sensor may be added in an invisible wavelength, and theretro-reflective surface may have fiduciary markings in color notvisible to the human eye, but contrasting in the invisible wavelengths,and the sensor may be able to recognize the retro reflection or itsabsence, thus being able to read human invisible fiduciary markings.Further, a user, interjecting objects, may create a disruption of thereflected invisible beam, and detection of such interruptions can beinterpreted by the system as commands for actions, including but notlimited to navigation in a virtual environment, launching of programs,manipulation of data, and so forth.

In addition, the user interface of the system and method disclosedherein takes into account the natural use of hands and objects by meansof “virtual” hands, simple auto calibration, and alignment with natural3-D vision, without the need to “transpose” like most pointing devices.It offers personal adjustments to actual eye positions and actualintra-ocular distance, as well as correct horizontal and verticaldisparity, correcting for inclination (for example, a user lying on acouch looking sideways) and changes during viewing and interaction.

By allowing fiduciary marks to be read on each scan line, not just fullimage scans, and using fiduciary markings that contain full locationinformation, a very fast feedback is provided, typically around 500-1000times the frame speed currently used by conventional methods with fullframe cameras, for example, including but not limited to, Kinect andSony PS3 EyeToy and 3D Systems. Those typically use cameras thatintroduce frame, shutter, and/or frame buffer serial delays, creatinghuman-noticeable latencies.

Further, by use of direct first-person view, the trip to screen and backcan be computed instantly and accurately, using a 500-1000× faster andnear zero latency instantaneous detection of first-person motiondetection. The projector and screen observing sensors are at the samevantage point and in a fixed relationship to each eye. Thus the degreeof uncertainty is greatly reduced because of not doing successiveestimations, each with both an error and one or more frame delay.Additionally, the system adjusts for changed head and eye position;i.e., vantage point detection, making adjustments for both horizontaland vertical disparity, head rotation, head rotation in response to fastmoving objects (such as following a ball, dodging or ducking a ball in amotion sport simulation or augmented reality games).

The construction of the device disclosed herein is simplified withintegration of only a few parts, low weight, and low power cost,enabling ultra-light, affordable Eye Stalks.

By employing screen corner cube array diversity (pseudo randomness,deliberate avoidance of excessive periodicity) the notion of specle andinterfrenece patterns can be vastly reduced.

Combinatorial efficiency of multiple primaries (four or more) withfeedback from the screen, from observed response is different from priorart that typically uses diversity as a deterministic solution,regardless of actual image and artifacts occurring. Further, theobserved response can be used to switch between various schemes to findoptimum trade off and varying them based on a combination of real timeinstantaneous observed screen response (e.g., speckle in certain areas,at certain sceen scan angles, with certain color mixes, at a certainbrightness). Furthermore, each primary can be adjusted for observedintensity versus intended intensity.

Multi layer screens (two and three ways) can combine multipleretro-reflective functions with specular (for virtual panning) anddiffusion (for illumination and pointing devices) and absorbtion (forhigh contrast suppression of ambient light).

Further, the system can be switched to allow any combination of 2-Dand/or 3-D projection within a same field of view and based on vantagepoint.

In some cases, rather than use a full head-mount system, the parts thatneed to be placed near the eyes can be implemented as “parasitic”clip-on users glasses frames and can connect to a mobile phone (smartphone) as a computing device.

These modifications and variations do not depart from its broader spiritand scope, and the examples cited here are to be regarded in anillustrative rather than a restrictive sense.

Therefore, comparing to the conventional approaches, the describedsystems and processes project relative and absolute positions of varioususers and elements faster and more cost-effectively.

What is claimed is:
 1. A method for image projection, comprising:providing a screen that includes a retro-reflective surface, wherein theretro-reflective surface is operative to reflect a light beam such thata cone angle of the diffused reflection of the light beam is at leastten times less than 180 degrees; providing a strobe light beam thatscans for the retro-reflective surface, wherein detection of thediffused reflection of the strobe light beam determines when a gaze of auser is directed to the retro-reflective surface, and wherein the strobelight beam is non-visible to the user; employing at least onehead-mounted projector that is mounted on a head of a user to project animage onto the retro-reflective surface when the user's gaze is directedto the retro-reflective surface in such that only the user is able tosee the image, and wherein the image is at least one of acomputer-generated image or a computer-fed image, and wherein two ormore fiducial markings that are located on the retro-reflective surfaceare used to stabilize and scale the projection of the image respectiveto the user's gaze; and when the user's gaze is detected to beundirected to the retro-reflective surface, ceasing the projection ofthe image by the head-mounted projector and pausing one or more movieswhen it is included in the image and the user's gaze is undirected tothe retro-reflective surface.
 2. The method as in claim 1, wherein theat least one head-mounted projector includes a first projector and asecond projector, the first projector is mounted on one side of theuser's head, the second projector is mounted on an opposite side of theuser's head, and wherein employing the at least one head-mountedprojector to project the image onto the retro-reflective surfaceincludes employing the first projector projects a beam onto theretro-reflective surface such that the beam reflects with a firstretro-reflective angle to project a first projector image, and employingthe second projector to project another beam onto the retro-reflectivesurface such that said another beam reflects with a secondretro-reflective angle to project a second projector image, and suchthat the first retro-reflective angle and the second retro-reflectiveangle are each sufficiently small that-each eye of the user can only seea corresponding one of the first projector image and the secondprojector image at a particular distance from the user to theretro-reflective surface.
 3. The method as in claim 1, wherein employingthe at least one head-mounted projector to project the image onto theretro-reflective surface includes projecting lasers onto theretro-reflective surface.
 4. The method as in claim 1, wherein employingthe at least one head-mounted projector to project the image onto theretro-reflective surface includes projecting highly collimated LED lightsources.
 5. The method as in claim 1, further comprising employing atleast one of a camera or sensor that is mounted near the at least onehead-mounted projector to adjust the image.
 6. The method as in claims1, further comprising employing at least one camera that is mounted nearthe at least one head-mounted projector to observe an interaction of theuser with the image.
 7. The Method of claim 1, further comprisingemploying a plurality of edge fiducials on the retro-reflective surfaceto determine a location of the screen relative to the gaze of the user.8. A system for image projection, comprising: a computing device that isarranged to provide image information; a strobe light beam that scansfor a retro-reflective surface, wherein detection of the diffusedreflection of the strobe light beam determines when a gaze of a user isdirected to the retro-reflective surface on a screen, and wherein thestrobe light beam is non-visible to the user; and at least onehead-mounted projector that is configured to, based at least in part onthe image information, project an image onto a retro-reflective surfacesuch that only the viewer can see the image when the user's gaze isdirected to the retro-reflective surface, wherein the retro-reflectivesurface is operative to reflect a light beam such that a cone angle ofthe diffused reflection of the light beam is at least ten times lessthan 180 degrees employing the at least one head-mounted projector thatis mounted on a head of a user to project an image onto theretro-reflective surface when the user's gaze is directed to theretro-reflective surface in such that only the user is able to see theimage, and wherein two or more fiducial markings that are located on theretro-reflective surface are used to stabilize and scale the projectionof the image respective to the user's gaze; and when the user's gaze isdetected to be undirected to the retro-reflective surface, ceasing theprojection of the image by the head-mounted projector and pausing one ormore movies when it is included in the image and the user's gaze isundirected to the retro-reflective surface.
 9. The system as in claim 8,wherein the at least one head-mounted projector includes a firstprojector and a second projector, wherein the first projector is mountedon one side of the user's head, the second projector is mounted on anopposite side of the user's head, and wherein the at least onehead-mounted projector is configured to project the image onto theretro-reflective surface such that the first projector projects a beamonto the retro-reflective surface such that the beam reflects with afirst retro-reflective angle to project a first projector image, and thesecond projector projects another beam onto the retro-reflective surfacesuch that said another beam reflects with a second retro-reflectiveangle to project a second projector image, and such that the firstretro-reflective angle and the second retro-reflective angle are eachless than about 10 degrees, whereby each eye of the user can only see acorresponding one of the first projector image or the second projectorimage at a distance less than or equal to about 1 meter from the user tothe retro-reflective screen.
 10. The system as in claim 9, wherein theat least one head-mounted projector is arranged such that at least onehead-mounted projector can be adjusted to reduce the first and secondretro-reflective angles.
 11. The system of claim 8, wherein the at leastone head-mounted-projector is arranged to project the image byprojecting lasers.
 12. The system of claim 8, wherein the at least onehead-mounted projector is arranged to project the image by projectinghighly collimated LED light sources.
 13. The system of claim 8, whereinthe at least one head-mounted-projector is connected to the computingdevice, the computing device includes a central processing unit (CPU),the computing device contains a software instance running on the CPU,and wherein the software instance is configured to create virtual 2-Dand or 3-D images for viewing by the user.
 14. The system as in claim 8,further comprising a sensor, wherein the computing device is configuredto control the at least one head-mounted projector such that, durationoperation, the at least one head-mounted projector projects the image byprojecting a beam, and wherein the computing device is furtherconfigured to control the at least one head-mounted projected such that,during operation, an additional beam is added to the beam in aninvisible wavelength, wherein the sensor is configured to sense whethera retro-reflection of fiduciary markings in color not visible to thehuman eye, but contrasting in said invisible wavelengths, are present onthe retro-reflective surface, and further configured to read thefiduciary markings if the fiduciary markings are present on theretro-reflective surface.
 15. The system as in claim 14, wherein thesensor is further configured to detect interjecting objects between theat least one head-mounted projector and the retro-reflective surface,the computing device is configured to, in response to the detection ofinterjecting objects, control the at least one head-mounted projector tocreate a disruption of said another beam, and wherein the computingdevice is further configured to, in response to the detection of theinterjecting objects, provide commands for actions.
 16. The system as inclaim 14, wherein the software instance is further configured tocalculate a position and orientation of the user's head relative to theretro-reflective surface.
 17. The system of claim 15, wherein thecommands for actions include at least one of navigation in a virtualenvironment, launching of programs, or manipulation of data.
 18. Thesystem of claim 8, further comprising a plurality of edge fiducials onthe retro-reflective surface to determine a location of the screenrelative to the gaze of the user.